lano1106 OF V2 5.75.5G journal

[lano1106]

Cycle 24 (v2 cycle #6), day 3:

I have no idea what did happen today but it felt like Christmas. My trading system went crazy. It bought and sold dogecoin back and forth non-stop since this morning.

It made close to 300 trades. My monthly trading volume is now approaching $250K and my bitcoin balance has almost doubled.

This is definitely an exceptional day but that was definitely fun to experience...

[CatMan]

LOL, that was likely the hype from Elon Musk's post today about Dogecoin that lead to that.

[lano1106]

Cycle 25 (v2 cycle #7), day 2:

The doge pump followed by the xrp pump made my system trade like crazy.

My current monthly trading volume is now about 300K. It did peak at around 310K but now it deflates a little bit now that the madness is gone.

At its peak, I had a screen with a list containing over 400 trades for the last 48h...

It does bring some quality problems...

For one, I was thinking that this would now be the regular pace... and I was seeing myself reaching the next fee bracket which is at $500K in a week or so...

but what I did experience was an anomaly...

but with all that activity, the log file got incredibly bigger. I need to change my way of analyzing it. First time that it grew up because of an exchange glitch, it became 20MB big and it has been a pain to review it....

The last sequence of trading frenzy generated 70MB of logs... No way that I can go thoroughly over all of it...

I search for plan execution summary traces and I just pass over the whole thing when it was positive. I only go more deeply if something went wrong...

Yesterday, during some execution, I did spot a small anomaly. I did search like a beast the log to undestand what the problem was. I was looking for a bug and instead I found and fix 3!

It was hard to spot due to some lack of logging in that area. I had to make some assumptions. The funny thing is that the one bug that I was looking for, it is as if I was blind and I couldn't see it. It must have been in my face for at least 3 hours without me seeing it.

A return statement in the middle of a function that was making the code following it essentially dead code. Code that never get executed... It is weird I thought that the compiler would have warned me about that but it didn't... I'll review the compiler switch. If I activate some maybe I am going to catch some other errors like this one simply by recompiling the code...

Once you know what it was, it was kinda obvious. I knew that the problem was in that function but I just couldn't see the problem despite the code was in front of me... Could it be subconscious sabotage... this feels like that was something like that...

They were quite serious... By having squashed them, I wonder how it was possible that the system was performing quite well despite their presence...

I have become addicted to low latency... the latency reduction did make my trading quantity explode... I'm in the process of finding another way to reduce further the latency because despite doing more trading, I still miss a lot because my system isn't fast enough...

So on one hand, this is super thrilling but at the same time, it is a source of fear because more trading means more logging and that logging thing becomes unmanageable to a certain point.

It is better to make sure that all the serious remaining bugs are found and removed before letting the trading explode otherwise, this is going to be painful to manage.

OTOH, the last week has demonstrated that even in the presence of bugs, the system is still profitable because of the trading volume...

Update:

Another rare occurance bug fixed!

[lano1106]

Cycle 26 (v2 cycle #8), day 4:

Wow! It is not in my habit to be not journaling for a long period of time.

Things are going fine.

As funny as it may sound, on each new bug that my system stumble on, I am grateful for it as I see that as an opportunity to make my system better.

I am incredibly surprised that bugs keep showing up in a system that didn't increased much few some time. I'm adding more robustness and stability to it.

It start paying off. A constant stream of trades is performed by the system. This increase in activity, is causing a problem with analyzing the logs. They are too voluminous and I have a hard to stay up to date with checking them.

I'm not sure yet how I will address it but this is certainly an issue that is slowing me down.

but overall, I'm feeling the progress. I'm slowly sailing toward half a million in trading volume...

I have finally found a way where I can subtantially decrease my latency with the exchange...

but there is a catch... It cost between 3.5K to 4K per MONTH...

My initial reaction was f... this... I'm very far from even generating this amount of profit with the system as it is...

but the more I think about it... the more I feel like I should give it a try... That could be the last puzzle piece that will enable my system to great profit...

but thinking doing this is probably the most insane and crazy idea of my whole life...

The only thing that I know is that before jumping on this nuclear option... I need to squash out as much bugs as I can because if I'm overwhelmed by logs now.... They are going to be produced much more faster once the nuclear option is enabled...

Very exciting... things are unblocking big time

[lano1106]

Cycle 27 (v2 cycle #8), day 7:

I had this crazy idea to immigrate to the Cayman islands. Crazy because AFAIK, unless you are millionaire, they won't let you in...

but beside being a tough objective, I love the idea because they have no income tax. They have 0 Covid measures. Everything is open.

It seems like those benefits comes along with having a lot of money... People with money don't accept those stupid measures and are able to influence the politicians to stay reasonable.

Anyway, I feel like with my project about to breakout, this crazy idea is reachable and this adds an extra motivation to succeed.

I must have used OF for almost 8 months... but unless there is a new program that can assist me in my current endeavor (UMS, BASE or something along those lines), I think that I may stick with OF even longer than 8 months. The reason being that I feel like the current environment is extremely anxiogene. Using OF, helps me to stay cold headed and remain sane in all the surrounding insanity driven by fear.

Concerning my project,

I have been trying to solve a complex issue happening when a trade plan is initiated by my system. I have added some new code to address that issue. It did seem to work but with after few test drive of the new feature, I have discovered a bunch of malfunctions that occurs with certain cornercase scenarios. One of them did even made my system crash during a mini flash crash in the altcoins market yesterday evening.

Am I progressing or regressing? adding new code introduces new bugs. I guess that the answer is that I am progressing as I am testing the limit and testing the system. Every bug fixed makes the system better but the last run feels like a catastrophic failure. To the point where I didn't restart the system until I have fully understand what it did and that I fix those quite severe bugs.

and I feel like, it is harder to focus and concentrate on the tasks related to my project. I feel distracted or maybe I am procrastinating a bit...

Another topic that isn't smooth is the bare-metal server. I have finally committed to a bare-metal server. I did create an order to have one. The provider advertise a fast order provisioning. They say, usually up and running 40 minutes after you have made your order and in the worse scenario you have your server within 24h. Well, the order has been passed well over 48h ago and I still don't have the server installed.

This sucks. OTOH, even faster HW would have mean more damage from the bugs that I have talked about above. So, we could say that this is a bad event for a good outcome...

but what I conclude from those events is that I am in a small period of resistance. Stuff does not happen easily and naturally as they were.

I accept this state and I beleive that it is only temporary and things will start get better in the next few days.

[lano1106]

Cycle 27 (v2 cycle #8), day Off:

I fix the crash that I did report in last journal entry. I have a new feature that I have introduced in last build that simply just don't work. I have added some traces to have more insight into that. I suspect that the problem is very simple but I simply don't see it.

My bare-metal server is ready but this morning, there has been a flash crash in crypto which made my system trade like crazy.

and it did crash again. It doesn't seem to be the same problem than the previous one. I'm not sure exactly what it is but I am feeling stressed about the whole situation. There are a lot reasons for that.

When my system is crashing, I need to unwind manually all the open positions back to the origin accounts. Just that is not trivial because the system generates a web of trades from multiple concurrent threads. Figuring which fund belongs to which account is complex task and a bit painful This need to be done fast because by leaving funds in intermediary currencies, I run the risk of losing money. Sometimes it can do the opposite but since the trade hasn't been made with a long investment horizon in mind, it is a risk.

Next, I have just thrown the towel in checking all the logs. Just this morning 30MB of logs have been generated. I think that I have a fear of losing control but I need to let go. The system generates too much logs to be manageable.

So to summarize,

I have a crash to understand and fix
Positions to unwind

and I have a server that is waiting to be setup once my trading system is able to run smoothly for some time so that I can do that.

I should be happy to see all those things happening. I remember last summer when I was worried because there wasn't much trading activity. Now, I'm in the exact situation. There is too much to handle and everything is happening all at once...

I'll take a long breath, try to stay calm. Manage one thing at a time and most important let it go. Accept that reviewing everything is not possible anymore.

Update:

I think that I have fixed the problem with the new feature and I have found the cause of my crash and I have added several small improvements that I have found while investigating.

I did restart my system. Crypto markets did experience another important drop and my system started to trade like a maniac and it did crash again on some other new bug...

ah....

to be fair with my work, I have to say that the problem seems to be caused by the exchange server that behave erratically under heavy load... So I get reactions that my system did never experience. This is good, it is going increase its robustness...

[lano1106]

Cycle 28 (v2 cycle #9), day 1:

In the last 2-3 days, I have done a lot of very vivid dreams. Because I didn't journal about them immediately, I don't really remember the content but the simple fact that this did happen is an indication that changes is operating in my mind.

I did progress about the described situation in the last 2 entries. I'm not yet out of the wood, but I am seeing the light at the end of the tunnel.

My biggest burden which was the logs is in the process of getting resolved. I have found few simple tricks to format the logs so that I can extract easily the logs that are specific to a single trade plan.

I could improve this even more. I got a simple idea just right now as I was writing this entry.

[lano1106]

Cycle 28 (v2 cycle #9), day 6:

I am about to launch the new server. In order to set it up, I had to setup VPN software to connect to the hosting corp private network. By having a not so mainstream linux distro, it didn't work straight out of the box.

I had to become an expert in the protocols IPSec, L2TP and PPP in order to make that VPN connection work.

but I had a blast doing it as in a distant past, I was developing protocols and I had a lot of fun doing this so, my VPN struggle was fun. I had to dig into a bunch of TCP/IP RFC documents!

As I was busy with the VPN, my trading system kept working on the VPS and it did fine... I just spotted at least 2 glitches that I want to make a quick review of before installing the system on the new server because, the glitch occurrence will just happen much more often on the faster HW. I prefer to do the best to eliminate it to not see it happening all the time on the new server.

One of them is already taken care of. One more to go.

[lano1106]

Cycle 29 (v2 cycle #10), day 5:

I have finished installing my new bare metal server. I do have a very small latency improvement over the previous setup. However, the expected result of this improvement translating to an increase in trading activity has been very disappointing.

In fact, I have seen a decrease in activity. I don't think that this is related to my server switch. It is factors outside of this.

I have few more ideas on how to improve latency... Good ideas keep coming... I'll continue trying them until I find the right one...

[lano1106]

Cycle 30 (v2 cycle #11), day 2:

I am starting to be really deep into the rabbit hole when it comes to optimizing an ethernet card to get best possible latency.

I did explore 3 ways to improve latency. The first 2 may/may not have improved the situation but worse case scenarios, they didn't make a significant difference.

The third one, I have only explored half of it and it has already demonstrated that with it, I will save at least 2ms. This amount is huge.

Incidentally, the most valuable way, I have looked at it last. I wonder if it means something about my subconscious priorities...

I suspect that the last thing that I have discovered is what is used by my most serious competitors.

I expect that when it is implemented, not only, it is going to level up the playing board, my system should take the lead because of everything else that I have put in place for improving my latency...

[lano1106]

Cycle 31 (v2 cycle #12), day 1:

I haven't been able to do much more than where I was last week.

I have polished further my optimizations ideas and my reaction time has never been faster than it currently is but my competition is improving as well.

Despite all the effort to improve my system reaction time, my trading volume did melt. It did peak at around $370K sometime in February and now it is barely above $100K...

My numbers tell me that I am still 7-9ms too slow.

I will probably need a cross-connection to fill that gap. I cannot be 7-9ms faster just by improving my software.

But cross-connecting with CF is expensive. About $4K/month. so going for it is going to be a huge leap of faith.

Will the gain be big enough to pay off this huge expense?

I still have other ideas for optimizations... Also, a special niche idea that I had last fall that I didn't go up to completion... I am unsure of what I should be doing next... If I knew that going for the cross-connection would immediately fill my 7-9ms gap, I would probably go with it right away but I don't know.

The safest thing to do is to push as much as I can the software improvements to see how far I can go and also if I end up taking the financial risk to try out the cross-connection, by having the best possible optimized system will increase the probability of success...

Probably that the best answer is somewhere in the middle.

[lano1106]

quick post to share a small realization that I just made this morning.

with the combo OFv1+OFv2, I have now reached the 8 months of utilization but I am still far from having use OFv2 alone for 8 months straight.

However, I have no plan switching to another program soon. There is no other program right now that is tempting me.

If there was a new trading program, that would obviously be of interest to me... Beside that, I am waiting for UMSv2.

I have a copy of WM waiting for me but this one and any other social related programs is of no interest to me right now due to the current social conditions with the lockdown rules.

Also, I still have fears. I did overcome one last week-end that I did not talk about in the journal. It has almost succeeded in changing my plans but I have finally overcome it and did stick with my initial intentions.

Just in the last post, I did mention a fear of investing a big sum of money. Yes there is fear there too but also good reasons to be prudent...

So OF program still has some elements to work on. It is not useless yet to keep going with the program.

[lano1106]

Cycle 31 (v2 cycle #12), day 3:

Amazingly productive day today. I did finish 3 tasks in my todo list.

No trading by my system since yesterday... Definitely something that I need to look into. Hopefully, I do have plenty of very good leads about what to fix next to start getting results.

Update:

I have just finished a small side task. It was unplanned but a trade execution made me realize something that could be done to improve my system.

I had the idea, I just for it and did it.

What productive day. It has been a while since I have written so much code in a single day. It is very satisfying... No idea what is driving all the creativity and productivity but who cares, I'm surfing on that wave.

[lano1106]

Cycle 31 (v2 cycle #12), day 4:

It started as a slow day and it was a sharp contrast over how I felt yesterday. It was like my mind in foggy.

Fortunately, I have been able to shake that state out.

I have completed another refactoring that I did imagine during my daily walk yesterday. It seems to be working. There is not much different in terms of behavior or performance for having completed it but it is some fundamental preliminary work on which the next building block will be based on.

No need to deploy it on the production server as it won't change nothing in terms of result.

What I have completed yesterday, I did put it in production this morning. It appears that the CPU usage is a bit lower as a result but nothing spectacular did happen neither.

What I want to deploy next is some sort of extra metric that I am not measuring yet that will give more some extra insights into the markets. Another metric that I added yesterday is how many trades does my system miss because it is too slow and what are the value of those missed opportunities.

I got the first answer this morning. Because I did install it somewhere around dinner time, it wasn't data collected during a full 24h period. So during the recorded period, my system did miss 5 trades having a total value of 12 bucks....

I'm glad to have added this metric... I'm pretty sure that the first result isn't that representative of the true value of the missed opportunities but after few days of recording, if it is, I'll find something else to do with my time rather than trying to optimize something that can only generate 12 bucks of profits...

Once the next metric is installed, I should start working on optimizing my system performance.

Also some sad news to report... After having peaked my monthly trading volume around $370K, it did go back down below $100K... This is a bad news as I came back down to second fee step. Just one notch above what is offered to people having $0 volume... I absolutely don't want to go down below $50K...