Processing save table with AM or PM in file name - processing

Question

I have a project where I have data come in via the serial port every 15 minutes. I am using processing to read this data and save it as a CSV.
I would like for a new file to be created every 12 hours. However, when the file switches from AM to PM the entire row gets saved in the PM file (all the previous AM values)
How can I reset the table and start saving to a new file?
saveTable(dataTable, fileName());
dataTable.clearRows();
I tried this but it just clears the CSV file.
String fileName() {
String fileName = "";
String month = "";
String day = "";
int m = month();
int d = day();
if (d < 10) {
day = str(d);
day = "-0" + day;
} else {
day = "-" + str(d);
}
switch(m) {
case 1:
month = "-JAN";
break;
case 2:
month = "-FEB";
break;
case 3:
month = "-MAR";
break;
case 4:
month = "-APR";
break;
case 5:
month = "-MAY";
break;
case 6:
month = "-JUN";
break;
case 7:
month = "-JUL";
break;
case 8:
month = "-AUG";
break;
case 9:
month = "-SEP";
break;
case 10:
month = "-OCT";
break;
case 11:
month = "-NOV";
break;
case 12:
month = "-DEC";
break;
}
if (hour() >= 12) {
hour = "-PM";
} else {
hour = "-AM";
}
fileName = "SensorData_" + str(year()) + month + day + hour + ".csv";
return fileName;
}
Update: Code for collecting and saving data
void serialEvent(Serial myPort) {
if (myPort.available() > 0) {
String serialDataString = myPort.readString();
if (serialDataString != null) {
serialDataString = trim(serialDataString);
float[] sensorData = float(split(serialDataString, ','));
TableRow newRow = dataTable.addRow();
if (sensorData.length == 4) {
temperature = sensorData[0];
humidity = sensorData[1];
moisture = sensorData[2];
int packet = int(sensorData[3]);
if (packet < 10) {
packets = "00" + str(packet);
} else if (packet < 100) {
packets = "0" + str(packet);
}
String time = str(hour()) + ":" + str(minute()) + ":" + str(second());
String date = str(month()) + "/" + str(day());
newRow.setFloat("Temperature", temperature);
newRow.setFloat("Humidity", humidity);
newRow.setFloat("Moisture", moisture);
newRow.setString("Time", time);
newRow.setString("Date", date);
}
saveTable(dataTable, fileName());
}
}
}

Answer 1

In comments you've mentioned
Clearing after a save does not work as expected,
To clarify, what I meant is, if you call clearRows(), previous data will be erased. Saving before clearRows() should save previous data only, saving after clearRows() should only save current data.
I wrote a basic sketch and to me it looks that this works as expected:
void setup() {
// make new table, add 3 cols
Table dataTable = new Table();
dataTable.addColumn();
dataTable.addColumn();
dataTable.addColumn();
// add 1, 2, 3
TableRow newRow = dataTable.addRow();
newRow.setInt(0, 1);
newRow.setInt(1, 2);
newRow.setInt(2, 3);
// save to disk (expecting 1, 2, 3)
saveTable(dataTable,"test1.csv");
// print (expecting 1, 2, 3)
dataTable.print();
// completely clear table
dataTable.clearRows();
// add 4, 5, 6
newRow = dataTable.addRow();
newRow.setInt(0, 4);
newRow.setInt(1, 5);
newRow.setInt(2, 6);
// save again (expecting 4, 5, 6)
saveTable(dataTable,"test2.csv");
// print (expecting, 4, 5, 6)
dataTable.print();
}
(It's also nice that saveTable() appends data (and doesn't overwrite data) in this case.)
This is how I understand how/when data flows in your setup:
Arduino sends data over serial every 15 minutes. You haven't specified if the Arduino has a real time clock (RTC) and the code there uses it to only output data every 15 minutes on the clock (e.g. at :00, :15, :30, :45 past the hour, every hour). The assumption is there is no realtime clock and you're either using delay() or millis() so the actual time data gets sent out is relative to when the Arduino was powered.
When Processing sketch starts, it reads this serial data (meaning any prior data is loest). The assumption is there is no time sync between Arduino and Processing. The first row of data from Arduino comes at Arduino's next 15 minute (not Processing's) after the sketch was started.
The issue you might be experiencing based on your short snippet,
saveTable(dataTable, fileName());
dataTable.clearRows();
if it gets called in serialEvent() is that you'll loose data.
(Confusingly, it doesn't like you're calling clearRows() from serialEvent() ?)
One idea I can think is having some sort of event when the switch from AM/PM to then (first save any accumated data with the previous filename), then clear the the table and update the filename, otherwise (in serial event, save the data with the same filename).
A hacky approach is, once the AM/PM suffixed timestamp is generated to check if this suffix changes and only update filename/clear rows when this change occurs (e.g. manually "debouncing").
Here's a rough sketch to illustrate the idea:
Serial myPort;
float temperature, humidity, moisture;
Table dataTable = new Table();
String packets;
int packet;
boolean isAM,wasAM;
String tableFileName;
public void setup() {
textSize(14);
try{
myPort = new Serial(this, "COM4", 9600);
myPort.bufferUntil('\n');
}catch(Exception e){
println("Error opening Serial port!\nDouble check the Serial port is connected via USB, the port name is correct and the port istn't already open in Serial Monitor");
e.printStackTrace();
}
tableFileName = "SensorData_" + getDateStampString() + ".csv";
}
public void draw() {
background(255);
String sensorText = String.format("Temperature: %.2f Humidity: %.2f Moisture: %.2f", temperature, humidity, moisture);
float textWidth = textWidth(sensorText);
float textX = (width - textWidth) / 2;
fill(255);
rect(textX - 10, 14, textWidth + 20, 21);
fill(0);
text(sensorText, textX, 30);
// get an update date string
String dateStamp = getDateStampString();
// check AM/PM switch and update
isAM = dateStamp.endsWith("AM");
if(!wasAM && isAM){
println("changed PM to AM");
updateTableAMPM(dateStamp);
// update previous state for debouncing
wasAM = true;
}
if(wasAM && !isAM){
println("changed AM to PM");
updateTableAMPM(dateStamp);
wasAM = true;
}
}
public void updateTableAMPM(String dateStamp){
// saves current table (old filename): we're vaing data before the AM/PM switch
saveDataTable();
// clear rows so next 12 cycle starts fresh
dataTable.clearRows();
// update filename (for next save (serialEvent) to use)
tableFileName = "SensorData_" + dateStamp + ".csv";
}
public String getDateStampString(){
return new SimpleDateFormat("yyyy-MMM-dd-aa").format(new Date());
}
public void saveDataTable(){
saveTable(dataTable, tableFileName);
println("saved",tableFileName);
}
public void serialEvent(Serial myPort) {
if (myPort.available() > 0) {
String serialDataString = myPort.readString();
if (serialDataString != null) {
serialDataString = trim(serialDataString);
float[] sensorData = PApplet.parseFloat(split(serialDataString, ','));
TableRow newRow = dataTable.addRow();
if (sensorData.length == 4) {
temperature = sensorData[0];
humidity = sensorData[1];
moisture = sensorData[2];
int packet = PApplet.parseInt(sensorData[3]);
if (packet < 10) {
packets = "00" + str(packet);
} else if (packet < 100) {
packets = "0" + str(packet);
}
String time = str(hour()) + ":" + str(minute()) + ":" + str(second());
String date = str(month()) + "/" + str(day());
newRow.setFloat("Temperature", temperature);
newRow.setFloat("Humidity", humidity);
newRow.setFloat("Moisture", moisture);
newRow.setString("Time", time);
newRow.setString("Date", date);
}
// save data, but don't change the filename
saveDataTable();
}
}
}
Note the above isn't tested (so might contain errors), but hopefully it illustrates the ideas aforementioned.
(One minor note on packets (which I'm unsure where it's used): you can use nf() to easily pad a number with zeros (There are similar functions like nfc(), nfp(), nfs())).
Another option (similar to what I've mentioned in comments) is to use java utilities to call a function after a set time (e.g. the difference in time since the start of the sketch until either noon or midnight, whichever comes first), to then repeat at 12 hour intervals. You can check out TimerTask, or if your familiar with setTimeout in JS you can try this Thread based WIP setTimeout Processing workaround.