lano1106 UMSv2

[lano1106]

Stage 4, day 7:

This morning 2 different problems did occur on my server.

One of them did require from me to write tests to reproduce the problem so that I can finally make sense of it.

Back when I was a professional dev, the job was most of them mandating that we write systematically unit tests for any code that we were writing. For some reason, this is something that I have always hated to do. Therefore, since I am developing on my own, I take the freedom to skip this step...

However, by seeing that amount of time that I have spent lately for fixing bugs, I start wonder if that would not be a good idea to create more tests upfront before installing the new code into the production server...

I spent at least the last week fixing bugs non-stop... I guess that this can be qualified as high quality problems. That means that my system is active and busy and that I wrote a lot of new code lately... I got this burst of productivity lately... I guess that bugs come along too...

[MegaMan]

Your thoughts about tests remind me of my own experience.
I'm a junior dev currently and working on a prototype. There are zero tests now in my codebase, which makes me nervous sometimes. In my case it is not because I don't think it would be better to have tests, but because I cannot spare the time currently. Features need to be done, tests come later.

So I'm in a similar situation to you: writing new code, running code, and then fixing bugs.
Overall I think tests might have saved me a bunch of hours, but as I'm still a junior I'm also not very quick at writing tests.
Hope that your tests bring you the sought after stability and save you some time in the future 
MM

[lano1106]

thx for chiming in...

I guess that 1 man projects are manageable as we do. That is developing unit tests not before they are needed.

One thing that experience is giving is the wisdom that testing is a design goal. Some code is simply untestable because it consist of a monolitic blob...

Creating nice interfaces that interconnect all the subcomponents together makes it easy to test later since you can isolate a subcomponent by making it connect with mockup objects that their sole purpose is for testing.

I'm glad to see that this has become a second nature in me. My code is usually easy to test out of the box or with little change. When I need to change it for making testing possible, I see that as code improvement.

[MegaMan]

One thing that experience is giving is the wisdom that testing is a design goal. Some code is simply untestable because it consist of a monolitic blob...

Creating nice interfaces that interconnect all the subcomponents together makes it easy to test later since you can isolate a subcomponent by making it connect with mockup objects that their sole purpose is for testing.

That's good insight and valuable advice. Thank you lano1106

[lano1106]

Stage 4, day 10:

I have downloaded DMSI v3.4 yesterday... It is funny because without having listened to it, I felt like I did experience some TID this morning at my gym during the workout... I have no intention to give DMSI a try anytime soon. I have still too much to do on the UMS side to consider a switch at the moment...

that must be placebo effect then but it felt weird and good at the same time...

For the trading system project... I have spotted a flaw in the margin code strategy. The first iteration is working but it simplifies the reality... Not handling the situation 100% correctly does not appear to be super important (especially when this code has not been called at all in the last few days, but I felt that the flaw was important enough to address the exception case...

and this is what kept me busy in the last few days... Doing so made me feel like I am opening a can of worms... Some aspects of the code that were crystal clear and 100% predictable prior to the code update became a little bit more muddier... I don't like the feeling and I am reviewing the changes several times to convince myself that everything will be correct. On the bright side, I feel like at the end of the exercise, the code will end up being more robust. As a matter of fact, doing these multiple pass reviews did make me discover a silly bug that went unnoticed so far... We will see how things will be during my next server upgrade that could, hopefully, happen sometime over the week-end...

I did optimize a very solicited data structure in the system and optimization ideas keep popping. #1 priority is to speed up my system reaction time. All the last few ideas that I got were going in that direction.

I am pleasantly surprised to find out that I can still squeeze even more performance from code that I thought was mature and perfect...

[lano1106]

Stage 4, day 12:

I have completed the margin code fix. My confidence level in it is not 100% but maybe I am worried for nothing and everything is ok.

I was in a hurry to upgrade the server because I had 3 big performance improvements to try out...

Hopefully, those will finally make difference... Anyway, I was very eager to find out so this did motivate me to work late this evening to finish everything and upgrade the server... Now, I have to sit back and watch what will happen.

Update:
I got an horrible week-end. Some events did occurs that did hit my nerves in very hard way.

Hopefully, after having suffered from what did appear to be fate or some invisible power having pleasure torturing me...

I am finally coming out from this bad mood and I am retrieving the solution finding mindset...

[lano1106]

Stage 4, day 13:

My server new version has been running for about 24h... Not much activity came out of it... It made 1 trade and it went wrong...

I could be unhappy about the outcome but I am not. I interpret the event as luck because it was improbable events but because they did happen, it did let me discover a small glitch in the code.

I got plenty of new ideas to improve the performance of the code flowing in my head.... 2-3 easy ones that can probably be implemented in the next few hours and tested right away tonight!

[lano1106]

Stage 4, day 14:

I got a nice surprise this morning.

A $1000 budget trade plan incorporating the margin trading feature did generate a 4% profit in a 1 second interval.

If that was not good enough, there was still a glitch that made my system sell bitcoins at a $112 price tag. Fortunately for me, that was essentially a market order and it did not have a bad outcome since the market was very liquid.

but I am going to look into why this did happen. It should be very easy to fix.

Beside that, I am about to introduce lock-free structures into my multithreading system design. This is going to be the first time that I will be using lock-gree structures. I am aware that a big chunk of my system reaction time is spent waiting for mutexes. Therefore, I started to read my latest book acquisition on the topic. I must have purchased it sometime last summer... I have been reading it more intensely in the last 2 weeks and doing that is inspiring me good solutions to improve my system performance...

First lock-free structure intro run should happen sometime this morning and this spice up a lot my morning. I'm very excited looking what it will do to my system performance...

Also part of what I have done over the last week-end made me realize something very interesting.

The exchange advertised fee discount schedule shows that if you trade over 10 millions USD value in a month, maker fee becomes 0%. Reaching this 0% fee certainly opens up new opportunities but the goal is so far away from where I am now that this appears unreachable for now...

HOWEVER...

something that I have discover that is not advertised, it is the following:
Fiat only pairs have a different fee discount schedule than all the other pairs.

1. Maker and taker fees are identical.
2. Trading volume required to have access to the 0% fee is 1 Million...

THAT is still a lot of money but it is quite feasible to reach. Therefore reaching a monthly trading volume of 1 Million is going to be my new short-term goal...

I should be able to reach it with all the improvements that I have made. My system trading volume did flirt once or twice with $500,000 this year... A million is only double that amount... I can do it!

[lano1106]

Stage 4, day 18:

The last 2 margin trades did highlight some major design oversights. The last one made the server crash. I have spent the last few days correcting those oversights.

One oversight did break a major assumption and did require a lot of changes to support what was required to do to make things work the right way.

That was a pretty good exercise. I still have to complete reviewing the logs to make sure that the fixes that I have just bake in do fix everything or if there is more to do... but I am quite satisfied of the changes... Code quality of that important system area did improve in many small ways as a result

In parallel to fixing the margin trading code, I kept thinking and looking for ways to improve my system reaction time.

I am into something. The answer is Linux futexes or something around them... I think that with this idea, this might cut down by half my current reaction time... I am currently reading Ulrich Drepper 12 pages paper on the topic to clarify how this is going to be integrated in the system. But I need to put this project on hold. Priority is to upgrade the server with the required margin trading code fixes as the server has been down for 2 days now...

It is 2 days where I might leave money on the table and losing opportunities to let the server stumble into undiscovered issues...

[lano1106]

Stage 4, day 20:

I have just upgraded and restarted the trading server. My monthly trading volume did take a serious hit while it was down because I was in the process of fixing the bugs shown in the last run.

The system has been off for about a week and the trading volume went from above 100K to about 30K...

That sucks because that means that I am returning to square 1 for the fee discount... First discount is when you get above 50K...

I have discovered few other bugs between my last journal entry and this one. I was not done but it was very small glitches compared to what I have described in the last entry.

Ok, so now that I am done with fixing bugs, I can finally look into improving the performance. This should be very fun and exciting to do...

Hopefully, the new server upgrade will crank back up this trading volume fast...

[lano1106]

Stage 4, day 23:

I have completed my lock free futex based synchronization system. By itself, it is quite an accomplishment that I am proud of. This is definitely outside my comfort zone and something that I have never done before... I am getting a lot of satisfaction out of it...

That being said, the result is a bit disappointing. I was thinking that I was hitting the bottleneck of my system and addressing it would reduce by half the reaction time. It seems like I did an error thinking that. The last improvement gave me an improvement of 5-10 uSecs at most...

This is disappointing considering that I must have spent at least a week on this task...

I need to give the improvement some time to conclude anything. It might make a significant difference in profitability but I doubt that the speed gain is big enough for that...

I'll need to continue grinding and improve the performance but I am going to call it a day and wait until tomorrow to continue...

[lano1106]

Stage 4, day 26:

The day after the last journal entry, I got a new optimization idea. I believe that I have put my finger on something serious this time...

I have been working on implementing this idea non-stop since then... Hopefully, I'll be able to deploy the new version of my server later today... I am very eager to see the result...

Beside that, I think that some negative person that I have met yesterday at a dinner did taste the program auric shield action. We got some nasty confrontation and the end result has been that she decided to leave the dinner...

[lano1106]

Stage 4, day 27:

I did work several hours today to complete changing the logs all over the place in the codebase. This is a very mind numbing task. I must have hundreds of spots to convert. It is painful and long to do... I thought that I only had 2-3 hours of work before completing...

Hopefully tomorrow, it will be completed and I am going to be blown away by the result...

[lano1106]

Stage 4, day 29:

After almost a full week of effort, I have finally launch the optimized server.

The result was a total disappointment. Granted when I watch the logging window it seems to scroll much faster than it used (or perhaps it is placebo effect) but it doesn't change the timing of the critical code path.

I must confess that I am shooting in the dark to find the bottleneck of that code path. For some reason, my profiler (valgrind) does not work well with my program. (I got an idea while typing this entry. It might be because it is caught offguard by my usage of io_uring...)

I did figure that performance issue was coming from the logging library. The reason being that I measure 2 different input.

A: Which is the baseline
B: Does A + a little bit more validation and about twice as much logging than when doing A

Since processing B was taking about twice as much time than processing A along with generating twice as much logs, logging code was an obvious suspect.

On top of that, upon inspecting the logging code, I spotted 2 issues.

1. Sometimes C++ can be treacherous where when you manipulate small objects (ie std:tring), the code does plenty of expensive dynamic memory allocation without you being aware.
2. The logging library is a thin layer over sprintf. sprintf is pretty fast but it needs to scan the format string and make a buffer copy. If your string has no substitution in it, it is a wasteful scan and copy. About half of my logs are of that type

Therefore, I did the following:

1. 100% Eliminate logging related memory allocation
2. Create a fast-track path in the logging code for 0 substitution log strings
3. Deploy everywhere in my codebase the improved logging (this is what was long and painful to do)...

This is definitely a good improvement but it was just the wrong one for the current problem at hands...

Am I learning something here? I guess that I could have made a quick test to validate the concept before going all in the project and spend a week on that...

To add insult to the injury, I have commented out a simple function call (getName()) used to build one of the log string to simplify it. This simple change that took less than 5 seconds to make did improve B processing time by 10%!

I must succeed make my profiler work on my program... Otherwise, finding the bottleneck is like the proverbial needle in the stack search...