Hey ringp and tz1 (and anyone else) - I need your help in a discussion:

We are currently shipping OpenLogs with v1.61 installed. v1.61 is very stable, works very well at 57600bps. "Very well" means that is can successfully log at 57600bps without dropping characters across a 400k byte file. v1.61 only works with standard FAT16 SD cards with a 1GB max size. This will obviously become an issue over the next 12 to 24 months as microSD cards move up past the 2GB/FAT16 limit.

v2.21 is pretty stable, works with 16GB cards, but tends to drop characters at 57600bps.

The goal of OpenLog is not directory manipulation, or fancy ls listings, or file editing, it is logging. "No Think" logging. Just plug in OpenLog and it should log flawlessly.

So my question is this, are we headed in the right direction? I need help improving the overall logging speed of OpenLog so that it drops as few characters as possible. I am ok with dropping fluff like the baud rate menu, verbose errors, help menu, etc, even the command interface if it will help us log more characters, successfully, at 57600bps and eventually 115200bps with larger FAT32/16GB cards (with lots of data on them).

Worst case is a 8GB card with 2GB of pictures already on the card and the user throwing characters at a constant 57600bps. Should we strive to be able to log in this scenario? Or should we just disclaimer and say 'you can't log at constant 57600bps'?

I've been trying to work through this myself without much success. There's another user on the forum with the exact problem and I wanted to bring this to the attention of the dev(s) directly.

http://forum.sparkfun.com/viewtopic.php?f=8&t=20540

Basically both our OpenLogs are in a strange state where sending them anything over serial causes them to loop "12<" indefinitely. This happened to both of us after attempting to send a ctrl+z to OpenLog to enter command mode. I also know specifically for me that my SD card was filled with logs 0-255 but were all blank. I reformatted but I still get a looped "12<", even if I try sending regular ASCII from a test sketch that was working before I tried entering command mode.

Any suggestions? Our OpenLogs are now dead in the water from following the provided documentation.

Hello,

I received an openlog a couple of weeks ago, just got to play around with it. Upon power up, it goes to command mode (12>). I can't get it out of it, CTRL-Z does not work, it appears as if the firmware is outdated/broken. Is there an updated precompiled firmware available? instructions to transfer it to the logger?

Thank you very much,

In another project, I've placed a V_REG_LDOSMD in the same configuration as the MIC5305 on the OpenLog board, and Eagle's Erc has thrown an error.

I'd wondered why the BP pin was o/c, and indeed the Erc expects it to be an input. According to the Typical Application on page 1 of the MIC5305 device spec this pin (BYP - bypass) should go via a 0.1uF to GND. Looks like the schematic of OpenLog may have mistakenly connected this cap to Vout along with the 1uF that should be there. This seems to be in both the PDF and the Eagle SCH file.

(also posted to SparkFun forum)

Feature request: It would be pretty helpful to read out the recorded files content as ASCII (instead of HEX) avoiding disassembling the sd card all the time

Boot loader destroys itself when bootloading greater than 0x7800 (30,720) bytes.

Hi,

I think I've found a bug and hope it's not just because I don't know how to properly run and use OpenLog... ;-)

I'm using the new 2.2 firmware and did the following:

1. new SD card without config.txt file
2. connect OpenLog to FTDI cable
3. use screen to connect to it

What I encountered:
Although the standard baud rate should be 9600 as far as I know I had to connect
using "screen /dev/ttyUSB6 19200". In the menu "baud" still returned 9600 bps which seems
to be true as printing the command menu had a noticeable delay.

It might just be a power surge when the card first comes on, but every time I insert a card, it resets the OpenLog (it also happens with others - jpeg trigger, for example). I don't think it happened with my arduino to breakout setup.

This is probably not a problem, however I was thinking about dual drives (save to B when A is full, and allowing swapping A).

After multiple append and close commands, I see 0x1a in the file from that last append command.

As this is an EOF(end of file) marker, should this be removed from the file and a new EOF marker created at the new end of file ??

I was interacting with the openLog (1.61) and I found a strange behaviour:

If I send the hex code for return (0x0d) then the first characther of the next command is lost..

fputc(0x0d);
fprintf("new file.txt\r\n");

The answer is:

ew file.txt unknown command

Instead if I use:

fprintf("\r\n");
fprintf(new file.txt\r\n);

Then the file is correctly created. I spent quite a time trying to understand what was going wrong (lol) in the serial cable loosing chars.. anyway, any suggestion?

Customer requested directory remove command.

Hello,
I just wanted to ask if it is possible to use a baud rate of 38400?
My sender is fixed to this baud rate and can't be changed.

Thanks a lot in advance!

Sting

Would like to upgrade to the latest and greatest version, but get the following error during compilation:

core.a(HardwareSerial.cpp.o): In function __vector_18': /Applications/Arduino.app/Contents/Resources/Java/hardware/arduino/cores/arduino/HardwareSerial.cpp:96: multiple definition of__vector_18'
OpenLog_v2.cpp.o:OpenLog_v2.cpp:232: first defined here

Tried compiling with Arduino 0021 on both Windows 7 64 and Mac OSX.

Best,

J Tuhtan

I have a microSD to full sized adapter and the card is recognized and I can throw something into the DCIM directory, but it a Video/Share, not Pro, so although I appended .jpg it might not take.

Can anyone else try this?

Hi, We're using a bunch of these (10-15) as loggers to record RFID data from custom made readers. The RFID readers are sending hex bytes to the RX of an Arduino Nano with RTC attached which interprets the RFID data, adds a timestamp and sends it out over TX to the OpenLog (at 9600, essentially straight out of the box).

In most cases these have been working great, but we've got a couple of OpenLogs that have stopped responding completely. No blinking when they're given power, no response to grounding either the SPI pin or the GRN pin - as such we're not able to reflash them... There wasn't any apparent damage, they just stopped.

On inspection, it looks like the voltage regulator is still regulating down to 3.3v, but I'm not sure what to look for on the pins of the atmega..

Any ideas how/whether we can recover them?

I think you can enable USE_SLEEP - there will be an interrupt which will restart things to pull UART data. You can't go into deep sleep since you need the clock for the UART (and SPI).

Setting PRR=0xe9 will shut down the rest of the chip (timers - I don't know if any are actually used - the timed flush might need one) - if you shut off the SPI clock it says you need to reinit it.

Doing both should drop power to under 2.4mA when the serial line is idle.
You can get another 250uA by disabling brownout detect and save/restore(reinit) the SPI.

I'm having a problem with OpenLog and have been sifting through the code to see what's happening. Along the way, I noticed a couple things that I wanted to bring up.

File: main.c
Function: void newlog(void)

501c501
<   if(new_file_number == 255) 

---
>   if(new_file_number >= 255) 
504d503
<       EEPROM_write(LOCATION_FILE_NUMBER, new_file_number);
508,509d506
<   sprintf(new_file_name, "LOG%03d.txt", new_file_number);
< 
511,512c508,510
<   while(!fat_create_file(dd, new_file_name, &file_entry))
<   {

---
>   do {
>       sprintf(new_file_name, "LOG%03d.txt", new_file_number);
>       
515,517c513
< 
<       sprintf(new_file_name, "LOG%03d.txt", new_file_number);
<   }

---
>   } while(!fat_create_file(dd, new_file_name, &file_entry));
520c516
<   EEPROM_write(LOCATION_FILE_NUMBER, new_file_number++); //Add one the new_file_number for the next power-up

---
>   EEPROM_write(LOCATION_FILE_NUMBER, new_file_number); //Add one the new_file_number for the next power-up

Explanations

504d503

new_file_number is of type uint8_t, but do you really trust your compiler that much? I sure don't. I know in Intel x86 assembly, executing JGE is just as many clock cycles as JE, and I'd guess the AVR machine code is about the same for such basic instructions. So I doubt we'd lose performance with this change while making the code a bit more robust.

504d503

Why write to the limited-write EEPROM the same variable twice in one function? It's an edge case, so it's not going to happen often, but it seems excessive. Any time the log count exceeds 255, just reset the value in memory, tweak it while looking for an available log, then finally save that to EEPROM.

The Rest

Just refactored the loop structure a bit to make the code cleaner and potentially save a few clock cycles. Also, in the final EEPROM write, new_file_number++ would save new_file_number's current value to EEPROM, then increment it in memory. The code should have been ++new_file_number in the first place. I suspect OpenLog always stores the last used number in EEPROM and takes at least one additional iteration when creating a new log file on power-up to find an unused log file because of this. My refactor avoids this problem by properly saving the value because it's incremented before EEPROM_write is even called.

Since I'm not familiar with GIT, I'm not sure how to appropriately submit my changes, so I'm hoping the above patch file in this ticket is acceptable. Please let me know if you'd like me to submit it differently.

Request: Consider adding SPI, I2C, or Dallas 1-Wire versions of OpenLog. The code could be written so it's 90% the same, just with different Interfaces for I/O (i.e. An abstract class for I/O with implementations for Serial, SPI, I2C, and 1-Wire.)

From customer comments:

"read log00005.txt" shows the file contents correctly

"size config.txt" shows the file size correctly ("12")

then "read config.txt" errors out with "Failed to open file config.txt"

Size function must be leaving something open? Not sure.

It would be helpful to add a command option to display some sort of version information - compile time, if nothing else.

I've got my new unit fired up and logging, but the "set" command only handled baud rate - which didn't match the command description here.

Guess it is time to build from source....

From product comments (Timer Wolf):

"If I could change anything about the openlog I would give it the option to write/read it's settings to the flash card so they could be backed up, would not constantly write flash for file name incraments, could switched easily by swaping cards with two different pourposes, could easily be reset if bodrate etc was forgotten you could just format the sd card."

Brand new OpenLog (opened it yesterday). Wired it up, stuck in an SD card and figured it would work. Realized that the SD card I'd put in was FAT32 formatted. Got another SD card and formatted it with FAT16. Things appeared to be being communicated. Pulled out SD card mounted it on my desktop. No log file, no config file, nothing.

So I wrote a simple sketch for my arduino mega (multiple serial channels), read from the openlog serial and print it to the computer serial.

As expected the first character out of the box is '1'. Unfortunately that's where t success stops, the next character is not a 2 but some other character which never seems to be the same. Usually several other, apparently random characters are output, followed, inevitably by another '1' and another random character or three, followed by another '1', and so on.

As the '2' character is supposed to indicate that the SD card was initialized I can only assume that there's either something wrong with the way I formatted the card or there's something wrong with my OpenLog.

Just now I noticed that the arduino sketch has stopped recieving any data and the transmit light on the openlog is dim, but not off.

Currently, with an Openlog version 1.61 (no hw rev on PCB), and a 2GB SD card (brand unknown at this time), if I continually write data at 57600 baud with a few microseconds between bytes (around 5325 bytes per second), after 1,073,643,209 bytes the logging stopped and no further bytes were logged.

I am writing to the Openlog blindly, in that I am not using the TX (from the openlog) line, just the RX line.

My CONFIG.TXT file is
57600,26,10,1
and I am reasonably certain that the escape sequence had not been sent (the incoming data is noisy, and is never below decimal 50)

Customer requested overall boot time to be minimized. I'm not even sure what it is. Add elapsed time to debug mode so that we can watch it, then try to reduce it.

Like the title, it would be usefull if openLog is able to write a log file on the sd-card dumping into it working messages, faults, unexpected/malformed command, errors and status of it's operation (well, correlated to some sort of time-index, maybe configurable at some kind of errorlevel degree 0-log none, 1-more, 2-full operations and so on).

At least on the eagle flie visible in the free version - the part pin 5 labeled "AGND" is connected to ground on the schematic, but there is no trace visible on the top layer of the board between a nearby via (which would hit the ground plane the "free" version won't show) and the actual pin.

(If there are 4 layers, a VCC plane might also simplify the layout).

Need to add a feature so that when OpenLog powers up, the logger automatically starts logging.

While logging in NewLog or SeqLog modes, it may be needed to store the current buffer to the file, but while not leaving the mode.

Ctrl+z would cause the buffer to record, then exit log mode. Perhaps we should add Ctrl+q that forces a buffer record, but keeps the module in logging mode.

This is a drop-in arduino library designed for the SD shield from Adafruit. Supports both FAT16 and FAT32.

http://code.google.com/p/sdfatlib/

In my embedded application I see an echo of the command chars sent back to me.
Is there an option to run in half duplex mode and not get the echo'd chars.

Just get the response chars.

don

I recently reflashed my OpenLog from v1.1 to v2.0, but when communicating with it, I quickly found that I had no escape key sequence. Turning on debug mode and reflashing showed the following:

Text Settings: 115200,-1,-1,0
Len: 14
Settings determined to be: 115200,-1,-1,0

After a little digging through the source, I surmise what happened was my EEPROM was uninitialized with 0xFF values.

read_system_settings() is supposed to check for this and default the escape sequence to 26, but what I found was that instead of 255, it's value was -1, thus not being caught.

Serial.println(EEPROM.read(LOCATION_ESCAPE_CHAR),DEC);
Serial.println(setting_escape_character,DEC);

Prints:
255
-1

I think there may be an issue casting from 'byte EEPROM.read()' and 'char setting_escape_character`

If I change the setting global variables from char to byte, the problem is fixed.

Not sure what else this impacts though...

I am having issues with code instability after making any substantial changes to the source. I have made some simple changes, such as changing 57600 baud to 19200 baud, and that works fine.

My next task was to make a function to auto-generate numerically sequential log file names. The function worked fine generating the name, and i was able to print the new filename to the terminal ok. However if i then added a line of code to create that file (copied directly from the "new" command), then i would get weird behaviour from the OpenLog board. The code would compile, and load to the board fine. But if i tried to run my command, the board would crash and reboot. I assumed this was an error in my code. What was odd however was that other functions which were unchanged, and previously worked fine, also stopped working. For example "ls" would print out half of the disk info table (NOT the file list) and then reboot. Commenting out the offending line of code fixed the problem again. I tried moving the file creation functionality out of the "new" command and into a function so that i could call it, rather than duplicating it in my command, which resulted in the same weird behaviour. It appears very much like some sort of memory corruption issue, but the current code uses under 50% of the available program space, so i don't think it is a stack or heap overflow issue, but i don't know how to verify that. On the off chance that it was, i ripped out all the commands i wasn't using, so the code was now well below it's original size, but that didn't help either.
I don't really know how to progress from here, i expect there has to be something amiss in my toolchain setup to allow this to happen, but am unsure how to work out what it is.
I can provide the source that reliably reproduces the problem at my end.
Also during the process of trying to sort this problem out, one of my OpenLog boards has stopped responding to hails, and i suspect whatever has corrupted the memory has also broken the bootloader. is this likely?

I have a concern with this device when used to log small amounts of data very often. Have you addressed the problem of memory wear? FLASH memory, as in the SD card, can only take about 100,000 erase/program cycles before the memory could physically stop functioning. If you had an application that was frequently logging data couldn't this be a problem. When ever you modify a file in a FAT file system, the File Allocation Table is updated with the new file size and the Last Modified Data, etc. Wouldn't this section of the memory card get worn out? Have you addressed this issue? Does the FLASH controller in the SD card do any wear leveling on its own?

From a user comment on SparkFun - to minimize current consumption, give user the option to disable status LED (blue) that flashes with each character received.

I'm not sure if this will save much current, but it's worth looking into.

I comunicate with OpenLog by Putty Terminal and I post you my Input and Output

• 12>ls
• CONFIG.txt 11

• new file.txt

• append file.txt

• <~> read file.txt
• Hello!..

I insert in file.txt this string "Hello!" and I press three times Ctrl+Z to Exit from append function.
The logger see the first two Ctrl+Z as character and just the third is a command for it.
Infact inside file.txt there's "Hello!.." not "Hello!"

There is solution for this little problem?

Thanks

Optiboot is an arduino bootloaer that fits in 512 bytes

http://code.google.com/p/optiboot/

As the sf9domahrs project doesn't seem to be very active at the moment I decided to post this issue here. Sorry if it's out of scope, just close the issue if there's nothing you can do about it.

I'm trying to connect the OpenLog v1.61 to the Razor IMU with AHRS firmware. Setting both to 57600 bps works fine if I connect to each module using the FTDI cable and screen: The IMU nicely transmits the raw data and also OpenLog seems to work (config.txt was set to 57600 bps and also the baud menu shows 57600 bps as the current baud rate).

However, if I directly connect the two it doesn't work anymore: The data is not completely corrupt as what happens if the baud rate was completely wrong. I do see part of the data columns but it's riddled with a lot of strange characters...

Any idea what happens here? I've selected "Adruino Pro or Pro Mini (3.3V, 8MHz) w/ ATmega328" in the GUI...

Out of curiosity, I decided to experiment changing the bootloader on the OpenLog to Optiboot, freeing up another precious 1.5k or so programming space, and speeding up firmware updates substantially.

The change introduces some differences with existing OpenLogs, but the update can be performed in the field to older Openlogs if desired.

The process is fairly straightforward, but there were a few gotchas for me. First, the high fuse setting changes from 0xDA to 0xDE, reducing the amount of flash reserved to the bootloader. Second, the version of optiboot distributed with Arduino 0021 is flawed, and will not work loading sketches the size of the OpenLog firmware. See this thread for details:

http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1287435600

Downloading the latest version of OptiBoot from their project page and flashing that to the OpenLog works.

Once updated, the OpenLog will use the same settings as the Arduino Uno, and thus the Uno board should be selected when loading the OpenLog firmware inside the Arduino IDE.

It might be nice if bootloader/fuse settings initially shipped were listed somewhere in the Wiki in case anybody (heh...) somehow manages to screw up the settings and never wrote down what they were to begin with...

How does OpenLog perform with logs that are greater than 100MB? I need to create a rig to test this. It's a problem because to reproduce errors, we have to send a serial stream at 57600 for around 100 minutes.

We recommend you use a microSD card with little or no other data stored on the card. Using a card with 1GB of MP3s on it (or any other types of data) will degrade the performance of the card and therefore OpenLog.

Much to my initial frustration the code is missing support for all the standard baud rates, specifically I was wanting 4800 baud which used for standard NMEA from GPS.

Attached is a patch and compiled HEX file.
Simon

Background: The more data stored on an SD card, the longer it takes for the pointer structure to find an open spot on the card. This leads to long write times (allowed in the SD spec). These long write times are longer than what the buffer within the ATmega328 can handle.

Solution: We recommend you use a microSD card with little data stored on the card. Using a card with 1GB of MP3s on it (or any other types of data) will degrade the performance of the card and therefore OpenLog.

It looks like the code is only checking the state of the RX pin twice (seperated by 2 seconds), normal serial data could cause this to happen and would then reconfigure/lock up the device.

I would suggest monitoring the state of the pin continuously during the 2 seconds
Simon.

   int check_emergency_reset(void)
   {
    //Check to see if we need an emergency UART reset

    DDRD |= (1<<0); //Turn the RX pin into an input
    PORTD |= (1<<0); //Push a 1 onto RX pin to enable internal pull-up

    //Check pin 
    if( (PIND & (1<<0)) == 1) return 0;

    //Wait 2 seconds, blinking LEDs while we wait
    sbi(PORTC, STAT2); //Set the STAT2 LED
    for(uint8_t i = 0 ; i < 40 ; i++)
    {
            delay_ms(25);
            PORTD ^= (1<<STAT1); //Blink the stat LEDs

            //Check pin again
            if( (PIND & (1<<0)) == 1) return 0;

            delay_ms(25);
            PORTB ^= (1<<STAT2); //Blink the stat LEDs

            //Check pin again
            if( (PIND & (1<<0)) == 1) return 0;
    }

    // Stayed low the whole time
    return 1;
    }

In the fat.c routine for fat_delete_file, it appears that it assumes that the entire set of filename directory entries lie in a contiguous region on the disk but I don't think the FAT specification requires this.

So if you had a two non-consecutive cluster directory, and the long filename started in the first cluster, but the shortname was in the second, it would run off then end and continue "deleting" into whatever the next cluster is instead of linking through to the shortname. It won't create such an entry, but might not find one by (long) name properly.

Or is this condition handled and I don't see it or there is a specification or just every implementation honors "don't cross discontiguous sectors"?

I am using OpenLog to connect to a proprietary microcontroller and I'm logging data at 57600 bps. I have been having problems with commands requiring large delays (20-100 ms) between them, otherwise they get chopped up. That's fine, however I also ran a test to see what happens when the log fills up (I'm only recording to the same file each session). I am using a 1 gig microSD card from sandisk, so I expected it to fill up and complain at about 1gig, but this was not the case. At 511.4 megabytes (49.9999%) the OpenLog said "error writing to file".and exited append mode. It will also not allow me to append to a new file either.

Initially I thought it might have been a firmware bug (the OpenLog I received was 1.0), but when I updated to v1.1 I am getting the same problem. I have not tested any other capacity cards, but is this a part of the design rationale; that it will refuse to log after the card has reached 50% capacity?

avr-size: unrecognized option '--mcu=atmega328p'

This happens under Linux. It can probably be removed from the main makefile.
line 487 as this works for me:
ELFSIZE = $(SIZE) format=avr $(TARGET).elf

Connect OpenLog and set it to 57600 baud. Then set it to append to a file.
Send the contents (ascii) of main.lss (the listing file - 480k) over the serial port.
Send a control-Z to close the file.
Put the SD card in the computer and compare the result to the original.

When I tried I lost 3 sectors, i.e. a diff (or FC) shows gaps in the middle and the file size of the log is short by 3*512.

I've got OpenLog attached to an Arduino. Seems to be powered without VCC/GND connections. Needs more testing. Possibly just parasitically powering itself through the protection diodes on the ATmega328 on OpenLog board.

There is a issue when reading from files. The start and length values can not be more than 5 characters long (including the space between them). If you want read 20 characters starting from 100; eg 'read file.txt 100 20', the OpenLog starts output once the '2' is entered.

Hi,
I'm trying to log data from my Techedge wideband sensor controller. So far, I've been able to communicate with the Openlog from my PC (I'm using a MAX232 to convert 232 to TTL, also the card reader or card seems a little flakey occasionally) but I haven't been able to log any data from the wideband controller.
I have just read that the wideband controller settings are "19,200 baud, 8 bit, 2 stop bits". I've searched for likely settings in the firmware but no luck (using v2.21). Can Openlog read data with 2 stop bits or are there any changes I can make to achieve this?
Kind Regards,
Matt.