Upload speed through a socket

Hello, I the software lead of an FSAE team called Schulich Racing at the University of Calgary (We build a 2/3 size formula one car and compete at events). We are trying to use our mangOH to transmit vehicle data to our telemetry web application.

We have a MangOH Red with the WP8548 module. It states that its upload speed is 5.76 Mbps, but we are not able to achieve this. We currently just have the 100 MB sierra sim.

Here is what is happening:

We have a python server running on an EC2 instance which receives data from the mangOH through a C socket. We send a struct with a size of 120 bytes through the socket at 10 Hz (every 100ms). The socket tells us that the packets are being sent successfully, but some of the packets are being lost somewhere. On the python server, we only receive every third (or so) packet. When running the C code for 2 Hz (send every 500ms), there are no issues. The code looks something like this:

void send(struct Data val, int sock) {
   send(sock, (struct Data*)(&val), sizeof(val), 0);
}

COMPONENT_INIT {
  system("/legato/systems/current/bin/cm data connect");
  int sock = 0;
  struct sockaddr_in serv_addr;
  while ((sock = socket (AF_INET, SOCK_STREAM, 0)) < 0) {}
  serv_addr.sin_family = AF_INET;
  serv_addr.sin_port = htons (8000);
  while ( inet_pton (AF_INET, "**SERVER_IP**", &serv_addr.sin_addr) <= 0) {}
  while ( connect(sock, ( struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0) {}
  time_t startTime = clock ();
  time_t check = 0;
  int counter = 0;
  while(1) {
    check = ( float )( clock() - startTime)/CLOCKS_PER_SEC*1000;
    while(check >= 100) {
      struct Data val;
      //READ VEHICLE DATA INTO STRUCT
      sendData(val, sock);
      startTime = clock();
      check = 0;
    }
  }
}

Any suggestions would be greatly appreciated. Thank you!

Hi Justin,

As you are using TCP, I would guess it is not packet lost as TCP ensured reliability, instead the TCP ACK, buffer and re-transmission mechanism may causing delay and overhead.
Any chance you can switch and try UDP for this time critical data?

Also, you may try ping EC2 from WP to understand the latency and round-trip time performance.

Hope it helps.
Thx

Hi @justintijunelis,

This is a very interesting project that you’re working on :slight_smile:

I am curious about the requirements. Are you making automated decisions based upon this realtime data? Or are you monitoring it on a laptop and communicating with the driver at the speed of human decision making?

Since you’re using a TCP socket, it doesn’t really make sense for some of your data to be lost along the way. You should double check that send() is always successful. You could do something like this:

void mySend(struct Data val, int sock)
{
    ssize_t res = send(sock, &val, sizeof (val), 0);
    if (res != sizeof(val)) {
        if (res == -1) {
            LE_ERROR("send() failed: %s", strerror(errno));
        } else {
            LE_ERROR("send() returned unexpected value: %d", res);
        }
    }
}

I also wonder if you are actually sampling as often as you think you are. Try doing something like this:

struct YourCarData {
    // Your stuff here
    uint8_t counterVal;
}

uint8_t counter = 0;
struct YourCarData val;
while (1) {
    // Get data into val
    val.counterVal = counter;
    send(val, sock);
    counter++;
}

If you see on the server side that your counter values go 1, 2, 3, 4, 5… 254, 255, 0, 1… then you know that you are in fact receiving all of the messages, but you just aren’t sending as many message as you thought you were.

My guess is that you the Linux scheduler is sometimes busy doing other things so your program isn’t always running. Your timer code is a busy loop, so that’s not ideal. I think you should do something like this because it will let the process go to sleep:

const unsigned int operationPeriodMs = 100;
struct timespec startTime;
while (true)
{
    int timeRes;
    timeRes = clock_gettime(CLOCK_MONOTONIC, &startTime);
    if (timeRes != 0) {
        LE_WARN("Weird... couldn't get the start time");
        continue;
    }
    // Do your operation here
    struct timespec endTime;
    timeRes = clock_gettime(CLOCK_MONOTONIC, &endTime);
    struct timespec sleepTime = { .tv_sec = 0, .tv_nsec = operationPeriodMs * 1000 * 1000 };
    if (timeRes != 0) {
        LE_WARN("Weird... couldn't get the end time");
    } else {
        struct timespec delta;
        subtractTimespec(&endTime, &startTime,  &delta);
        subtractTimespec(&sleepTime, &delta, &sleepTime);
    }
    nanosleep(&sleepTime, NULL);
}

void subtractTimespec(struct timespec *bigger, struct timespec *smaller, struct timespec *result)
{
    // you implement this.  Make sure you never put negative values in result.
}

Thank you for the swift responses. The UDP socket method seems to be working very well, but I will try to work with @dfrey’s suggestions as well, ideally we would use a TCP connection so we don’t have to worry about implementing any sort of reliability measures on the backend.

As for your questions regarding requirements: No, we unfortunately cannot modify vehicle specs in real time as this would be against FSAE rules.

We currently have a web based system set up where the mangOH sends data (read through a CAN bus from our data acquisition unit) to a python server which formats data and does some math, we then forward that data to a Node.js server. This server communicates with a PostgreSQL database which logs our data for every time we turn on the vehicle (eliminating the inconvenience of having to plug in to the car to get our data). We also have a React based frontend that shows data vis, analysis, warnings, lap time +/-, and a bunch of other stuff.

We have communications on our car for radio, so we only communicate with the driver at the speed of human decision making.

We would love to be able to log at 60 Hz, but unfortunately we are on a pretty tight budget (Making race cars is expensive!). We were able to make a prototype for last year’s vehicle, but we are really doing our best to make a full-fledged version 1.0 this year.

If you would like to see more about our team, you can check us out at schulichracing.ca.

1 Like

Hi Justin,

Thanks for the info and link, https://www.schulichracing.ca/, indeed, it is interesting to know a MangOH board is embedded in your team’s racing car.


(Hope you don’t mind I pasted the picture here, I am working for the automotive team in Sierra so I am very excited)

As you tested and it works using UDP, I guess that means the code is handling the required data rate quite well.
In additional to David’s suggestion, TCP is “reliable” protocol and stream oriented, so it has to wait for TCP ACK, outgoing data is keep in buffer before the ACK arrival, so it is sensitive to network latency and may combine multiple payload into one. (this may cause you the impression of data lost?)
Tuning the socket option in code/kernel may improve but if the network latency between WP and server is too bad then it maybe difficult… so personally, I would first do a quick test using ping to see is that a bottleneck first.

Also, is the signal strength with tower good? that could affect the throughput.
Thx

The following was the ping output with a very strong connection (The connection usually hovers around 4-5 but may change depending on where we are, our competition this year is in Fremont, California).

86 packets transmitted, 85 packets received, 1% packet loss
round-trip min/avg/max = 701.694/806.612/2223.684 ms

We are trying to minimize latency so UDP will probably be the best choice.

P.S. That picture is the SR20 (Built for the 2018-2019 season), below is a render of the SR21 (We are currently in the process of building it).

Thanks for testing.

Yes, the ping time is not good… maybe the network route…

I would guess you are using AWS EC2 service? if yes, probably you can try other instance in different region (i.e. EU) to see any improvement.

I recommend using a Legato timer instead. It saves you the hassle of having to mess around with struct timespec and it gets your code out of the COMPONENT_INIT and into an event callback, so you can make use of other callbacks in the future too (if you never return from your COMPONENT_INIT, your event loop never runs).

So, in your COMPONENT_INIT, you would

le_timer_Ref_t timer = le_timer_Create("NameOfTimer");
le_timer_SetIntervalMs(timer, 100);
le_timer_SetRepeat(timer, 0 /* 0 = repeat forever */);
le_timer_SetHandler(timer, DoTelemetry);
le_timer_Start(timer);

Now, when you return from your COMPONENT_INIT, the event loop will start calling your DoTelemetry function every 100ms.

static void DoTelemetry(le_timer_Ref_t timer)
{
    struct Data val;
    //READ VEHICLE DATA INTO STRUCT
    sendData(val, Sock);
}

Just make Sock a file-scope variable by declaring it before any of your functions.

You can still open the socket, etc. in the COMPONENT_INIT.

Thank you everyone for your excellent advice. It’s very much appreciated! Midterm season has rolled around so I have been a bit busy, but I will be sure to fix up the code and get going on other parts of the project.

I have temporarily deployed the frontend so you can have a look at: http://schulichvelocity.com.

We have quite a bit left, but we still have 4 months to competition. We’re planning on adding custom line/scatter plots, along with some other datavis. Also, we will be making a historical data section to download csv data of runs, and plug it into some Matlab for further analysis. We’re considering making an app in the future with a 3D model of the vehicle updated by actual data too.