GitHub Gist

Used to create Random Walk, time-series CSV files larger than the available RAM.

Creating files larger than RAM

In a previous post I included code and a webpage to create random-walk time-series CSV files. Those code samples are only capable of creating CSV files that fit in memory; that is to say if you have 4 GB of RAM you will only be able to create CSV files less than 4 GB in size (after accounting for the operating system and other processes).

Backpressuring in Node.js streams

Node.js streams can account for this build-up of data that can occur in streams using a system called backpressuring. This allows a stream to send a “slow down” request to the source of the data.
By modifying the original CSV generation script, the code will only generate values as quickly as the machine can write them to disk. The key line is the pausing of the stream:

1
2
3
4
5

// if false, there is backpressure from the stream - stop sending new data
let ok = writeStream.write(`${cells.join(',')}\r\n`,);
if (ok) {
    return write();
}

if the writeStream.write() returns false, then the write buffer is indicating a data build-up and would like the stream to be paused so it can “catch up”.

Usefulness of the script

This script allows the creation sample CSV files any size - much larger than the RAM available on the computer running it. This is useful for creating large CSV files to stress-test programs that process files larger than the memory available.

Generator Webpage & Node.js Script

Sometimes, I just need some sample CSV data to test visualisation tools. I put together a simple Node.js generation script for creating larger datasets, and a simple webpage for small datasets.
The command-line script and the webpage allow you to create CSV multi-dimensional datasets that are gigabytes in size and decades in length, or just a small single series that fits in your clipboard.
The datasets are also Random Walks (vs random noise), which gives a better idea of how a chart will display real time-series data.

CSV Datasets

CSV datasets can be a useful way of storing time synchronous time-series data in a format that stands the test of time. CSV files don’t require bespoke programs to open or special know-how to unpack. Any CSV files created today will still be editable by engineers in 20 years without having to maintain legacy programs or databases, which is important when maintaining physical assets with multi-decade lifespans.

Hanging Christmas Lights

This is my first Christmas in my house, and when I came to hanging Christmas lights I discovered that there are no existing hooks on the front wall. I didn’t want to start drilling more holes into the brickwork so I decided I’d use an adhesive to stick some hooks on.
This seemed like a perfect opportunity to use my 3D printer to design and make some clips.

The Idea

I wanted a clip that would stick to the wall with double sided tape, and would hold the wire of the lights securely. I envisioned a U-shaped clip and set about on trying out the first iteration, knowing I’d be able to modify the design and print another in a matter of minutes.

The Iterations

I used Tinkercad to create these designs in 10 minute coffee breaks throughout the day. It was my first time using Tinkercad (over other 3D modelling software), and it was perfect for the speed and simplicity this mini-project needed.

First

The first clip was to work out the approximate size needed, as well as the thickness. It was clear that this design was too big and I imagine a gust of wind could unhook the wire.

Second

I added a triangular clip to prevent the wire from slipping out. This worked well, but the shape / thickness of the clip made it far too stiff to easily slide the wire in.

Third

I added a circular loop at the base and make the clip thinner to lower the tension. This worked well and I was happy with this design.

Printing

It took a couple of hours to print all the clips, but there were no problems using my Ender 3 Pro to print a batch of 8 clips.