Subliminal Talk

Stage 3, day 32:

I did work the whole night last night. Tonight, I'm really tired and I'm about to go to bed.

The main reason behind me working all night is that it started working. I got a bunch of small annoyances and glitches. but it works very well. I'm super satisfied to see the fruits of all these efforts in the last months finally give something. That was very exciting and I didn't feel the fatigue and see the time fly.

Due to some bugs, my trade sequences did interrupt leaving me with a bunch of NANO coins. No problem, I did use one of my dynamic execution strategy to convert those NANOs back to bitcoin and it did a small profit at the same time.

I added a lot of polish and I did squash many small bugs and I'm starting to see the potential of my creation but I won't let it go freely before going to bed. I'm too tired and this creates a potential of errors that could be costly. Tomorrow morning, with a hot coffee, I'll put the final touches for release V1 and give the beast some leeway to show me what it is capable of.

Either tonight or tomorrow morning, I'm starting Stage 4.

Stage 4, day 2:

I did one more test run yesterday. It did encounter a bunch of minor issues. Even if the trade did fail, the experience was very successful as it did open my eyes on a bunch of possible events that can happen that I wasn't aware before the run.

The run logs did also allow me to squash the last few remaining bugs.

The test run did lead some of my funds being stuck again in the PAXG/XBT market. I did refine further my execution engine to avoid that situation in the future. That is probably the worse that can happen. I would prefer taking a small loss than being stuck in the book for many hours.

I have a faith that the wind will come favorable soon and will let my order to complete but I'm stunned how illiquid and slow this market is. It is painful... I might possibly create some sort of pair blacklist that I would exclude to avoid the current situation. But this is a last resort only solution. The one to blame in the current situation is that my execution engine wasn't clever enough yet.

The bright side of the situation is that it allows me to take a break from the trading project to give some attention to my digital marketing biz. I have a small email backlog that did accumulate in the last few days while I was sprinting to the set goal with my trading project.

I'll need to come back later about the impression that stage 3 left me. I'm a bit in a hurry right now... but I'll say this. Overall, it is great. My project is moving. I'm reaching my goals. I feel like I'm on the path on something that could change my destiny. Nothing less than that... So, I'm very happy of stage 3...

and I'm sure stage 4 will even be better...

So stay tuned... I may have other impressions about stage 3 later and maybe I'm going to write down my expectations for the new stage4.

stage 3 recap:

Honestly, I have been so busy so focused on my project that I didn't really have the time to stop and think about what BASE was contributing to the experience.

One striking point is the script: Dream Your Best Solution

It seems that I'm executing this one very well. If I read back the last few pages, it did happen many times that I did report having found a solution after having slept on it.

Extreme focus is extraordinary. I see something weird in the application logs, I won't stop looking for an explanation. Doing that did pay off very often in finding out a bug. Or it did improve my understanding with the application domain that I'm working on.... Weird to say to I'm starting to be intimate with trading infrastructure and a lot of its surrounding concepts.

sometime, I look at some code... and I feel something is wrong with it but I cannot tell what exactly. That feelings makes me very uncomfortable. I kept looking at the code... and I did find something that could have wreak havok down the road if left unaddressed. It is as if I had a sixth sense for that sort of thing...

Something is changing into me... Maybe it is because I'm really absorbed into what I'm doing. I'm feeling some sort of urgency to succeed and this serve as a strong motivation to keep focus on the goal.

Bottomline, I'm discarding the small details... what might be interesting but not crucial to reach the destination. Something like thinking what stage 4 will contribute to my success. I do run it and follow the instructions but this time... I'm just going to put it on cruise control and trust it that it is going to do what it has to do while I'm focused on getting the job done. You could call it some sort of task delegation. (I think that there is a script for that... but I don't have much employees to delegate to yet... BASE is, in a certain way, something that I can delegate the task to give me the right mindset to succeed...)

Stage 4, day 6:

I took a small break to take care of all the accumulated unrelated tasks to the trading project.

The excuse for the break was that I had some fund stuck in PAXG that I wanted to transfer back into BTC at a specific price to not realize a loss. Not sure if it is Thursday or yesterday but the wind has changed direction. BTC took a dive and gold did soar and that was what I needed to exit PAXG.

In the meantime, I did integrate a set of improvements into the code and I should be ready to do a second test run soon.

One of the task that I wanted to accomplish is to compile the code on a webserver that I have access to. That alone has been a fantastic journey (not 100% completed yet).

The challenge of that task was that my code is built on top of recent libraries while the server is running on software that is almost not supported anymore (CentOS 6 with kernel v2.6.32, glibc 2.12, gcc 4.4.7 and openssl 1.0.1e)

I needed to recompile openssl 1.1 (took the latest one)
compile a JSON lib
compile a websocket lib

On openssl, the installation script complain that the old compiler isn't supporting 128 bits variables, therefore, it will turn off some features. Oh well. What can I do.

Here comes the JSON lib turn.

This is where I did put my finger in the sw dependency hell gears.

Installation scripts refuse to run unless I upgrade autoconf. Start to download autoconf. Then this one complains about automake.

Automake want a libtool upgrade and I had to upgrade texinfo and help2man as well for some obscure reasons.

Ouf, I have been able to close back the pandora box for few moments.

websocket lib building has been painless.

Now, I reach the part where I try to build my own code. On the first compiled unit, the compiler stops and report errors.

It complains about something that I do in my code that have been valid for some time but not at the time that this compiler has been built. At first, I say to myself I'm going slightly change my code to satisfy this old compiler. Probably the fastest way to address this.

The second reported problem is more serious. Some glibc function that I call isn't in the old system glibc version. From prior experience, I know that it is very perilous to manually upgrade glibc on a old system. The reason being that every single program on the system is using that library. If you do a manipulation error with the upgrade, you essentially brick the box because nothing will work anymore.

A safer approach is simply to create a private version and just link the process that needs it.
https://stackoverflow.com/questions/8471...ingle-host

So, I download the latest version: 2.31

Installation script complains and tell me that I'll need the following component upgrades:

- binutils >= 2.25
- gmake >= 4.0
- bison >= 2.7
- python >= 3.4
- gcc >= 6.2

That is the real dependency hell that I'm talking about....

I took care of them one by one. The crunchiest one was the gcc compiler (9.3) which also has its own dependency requirement:

- gmp
- mpfr
- mpc

After hours of compiling and addressing dependency requirements, I finally end-up in succeeding compiling the latest glibc version (2.31) with the latest compiler (9.3) on a 10years old barely supported OS!

Moment of truth, I try the new glibc. At runtime, all it does is print out an error message:
FATAL: Kernel too old

Are you serious?? This after having spent a full night working on building all the required dependencies this is only after all this trouble that you tell me that I can't run this thing on my OS??

A small check by the installation script and letting me know about this upfront would have been appreciated...

Ok, after a small google search, I did figure out that the last glibc version that still support my kernel is 2.25 (2017) which is still a giant leap in terms of performance, functionality and bug fixes over the current 2.12 version on my system.

Ok, I start to compile 2.25 and its build is abruptly interrupted in the middle. After some research, I discover that a patch has been created for that problem:
https://sourceware.org/git/gitweb.cgi?p=...905638b797

This is unbelievable. How can you release software without even verifying that it, at least, compile fine out of the box... Anyway, I apply the patch and restart the build....

Finally, I got this upgrade:

My attempt to replace glibc only for my app did work. I did upgrade:

# /lib64/libc.so.6
GNU C Library stable release version 2.12, by Roland McGrath et al.
Copyright © 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.4.7 20120313 (Red Hat 4.4.7-23).
Compiled on a Linux 2.6.32 system on 2019-04-09.
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>

to:

# /home/juicingf/dev/glibc-install/home/juicingf/lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.25, by Roland McGrath et al.
Copyright © 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 9.3.0.
Available extensions:
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>;.

It works so well. That I think that I'm going to push my luck and create some rpms to make a global upgrade of glibc on the whole box. This could give some substantial kick to performance on the whole box. I'm going, of course, test my rpms on a crash test dummy VM. The last thing that I want is to brick the server... This is a real possibility when you mess with glibc... (I know because I did it in the past...)

Bottomline, This is very satisfying experience. That was a very hard challenge. I did share my project on the websocket lib mailing list when I had issues with it and openssl. The lib creator simply told me that what I was trying to do was impossible. Hopefully, all the nitty gritty details give some sense on the toughness of the task.

To say the least, it was ambitious. So happy to have made it run. Despite being a hard task, it has felt fun and easy to me... Probably some BASE induced state of mind did help me in succeeding...

Stage 4, day 12:

I did finish to compile my program on the server machine.

I did test #3:
Plan execution did failed because of bitmask setting error. I was ORing the new value before clearing the underneath bits.

Test #4:
It looks like it went horribly bad but actually, objectively it was very successful as it did allow to discover a bunch of errors. I still have one mystery to solve but so far:
1. Not synchronized system clock (was ahead by 20 seconds) was making the program timing out when calculating elapsed time between now and some timestamp coming from the exchange.
2. A glitch in the strategy code was making the order getting cancelled and reset at the same exact price.

Stage 4, day 13:

Great success this morning. I have finally solved the remaining mystery that did obsess me for the last 36 hours or so.

It wasn't easy at all to find out. The bug did occur in a 2 seconds window. About 500 lines of logs are covering the events that did occur within those 2 seconds. And somewhere in those, something not very obvious is off. This is a perfect example of the classical finding a needle in a haystack.

Usually, I do inspect core dumps which make solving these kind of issues trivial. In this case, it wasn't available. Despite having root access to the server, I'm discovering surprising side-effects of being a VM.

1. It is impossible to change system time. It is protected by some Linux security feature that I'm not very familiar with. Capabilities. The init process. which is the ancestor of every processes on the system, possess a capability bounding set which all the system processes inherit. This capability set is missing the CAP_SYS_TIME capability. There doesn't seem to have any possible workaround that. I did ask the host machine admin to synchronize the time. He did (I wonder what he did thought of my request. It must be pretty unusual that one of his users complains about the clock being 20 seconds off....). It is amazing, in order to not disrupt anything running that expect continuous time, the time adjustment is performed by slowing just a bit the time. I did follow the adjustment. It took roughly 12 hours to lose a 20 seconds advance. Bottomline, I did adjust my code to take protective measures and not assume that the time is good.
2. Even with root access, it seems like it is impossible to enable core dump creation... I did work with such limitations. The only tool you have in that situation is very carefully crafted logs. That is it.

Back to my mystery. Despite looking at the logs, I couldn't find the problem. I started to reformat them to make them easier to look at and understand the flow of events. I started to annotate them. This morning, I was about to give up. I was in the process of creating a plan B. Such as adding a bit more logs that could help to catch the problem. Despite having succeeded making the software run on the server, rollback to my dev machine where I can have core dumps. and so on.... But before giving up... I did another annotation pass... And with this... It did allow me to spot 1 message among the few dozens messages exchanged back and forth... And that message was out of order, actually, it was missing... Upon that realization, I immediately understood that it was the reception of that message at an unexpected time that did crash the process. Without looking at the code to get confirmation, in my head just with the realization, it was game-over. I got the bastard. I did smile and laugh while I am alone at my desk... Best way to start the day....

Ok, now that this is out of the way, lets get back to the regular program and lets finish the money making software so that I can fully enjoy my quarantine...

Next priority once I stabilize the code base and have a satisfactorily working prototype will be to perform some refactoring.

Two of my modules are getting pretty big (1600 and 2500 lines of code). This starts to make them pretty complex and hard to maintain.

Breaking them into smaller components would definitely help in managing them.

Stage 4, day 14:

I'm done with all the tasks that I had to complete before being ready for the next test.

The program is now running. I'm just waiting for the next opportunity to see how the program does with the last batch of corrections.

Update:
I got my first trading opportunity traded. I'll need to increase the execution aggressiveness. I still got stuck in PAXG. The whole execution took again another 3 hours. Ideally, the whole thing should be completed in less than a minute.

On the positive side, the program completed the plan without any manual intervention. My creation is now able to fly on its own!

Now, I'll do a log postmortem analysis to see if things went as expected or if I'll still see rough edges in the execution that needs some polishing. By glancing quickly at the log, I cannot tell if the trading operation has been profitable or not... (not good).

The reason is that part of the way that the program work, it is adding up any residual amount found in the intermediary currencies balance. It turns out that due to one or two past unsuccessful tests, I had a significant amount in PAXG (about $60 worth of PAXG) before the last execution that got included. So of course, I have more BTC after than I had before. but the result got greased by the extra $60 baked in. I'll need to analyse the logs to get an accurate picture.

Here is where I think this is going: I'll probably find out that I have a X average execution success rate. The key to profitability will be to know what is my average loss so that I know where to raise the profitability bar used to determine if I pursue an opportunity or not when an opportunity is detected such that my average profit is higher than the average loss.

There is some code that I added for polishing that made me make a nice discovery. The whole execution game is to modify the price you are asking or bidding according to market conditions. Usually what happens, is that you allocate a fixed amount for a given order. When on the selling side, you can play all you want with the price, your input won't change. When buying, if you want to keep your input fixed, every time that you change the price, you need to tweak the volume as well since the input is the order cost.

First time that I did test that code, I did found that I could return to the initial price, yet the volume was slightly higher than the initial volume....

the discovery is this: You can tweak the volume to take advantage of how the exchange is rounding the cost so that you can get some extra crypto fractions for free. ie (not the exact numbers but you'll get the idea).

buying 0.00200033 BTC can have the exact same cost than buying 0.002 BTC. Granted, to only get a USD cent worth of BTC would require thousands of transactions but I think that I'll optimize my code to take advantage of that rounding side effect. I guess this is a known trick among the experts that I have just rediscovered... or maybe they just don't bother with it since it is such a small amount... I cannot stop thinking about the old 'Superman II' movie of my childhood where a programmer guy tweak his company pay system to have all those fractions of cents deposited in his account... He end up arriving at the office in a red convertible sports car the next morning... I cannot help but categorize my discovery into the same category of 'optimization'....

PPS: Actually, the superman movie memory (maybe it was in the first superman movie... I cannot tell for sure). It is strange that for a lot of things in my memory starts to get blurry as I get older but this particular memory from my childhood. I was maybe 6 years old is crystal clear in my mind and is actually inspiring me today in my BASE journey... Is it coincidence???

Stage 4, day 15:

Idk, if I am hallucinating or not but there is one mysterious script inside BASE. It is "Psychic Ability To Get Ahead As A Successful Entrepreneur"

I feel like I am receiving benefits from it. Again, idk if it is possible but it seems to me like some cosmic or universal wisdom is whispering in my mind some of the very cool ideas that I am thinking for improving my system.

I am receiving very good suggestions from something (maybe my own subconscious or some external influence) that is making success inevitable because what I come with is so amazing...

Stage 4, day 16:

I did complete all the needed tasks for the fifth test.

but something went very wrong. I think that I have a memory corruption issue somewhere. I kinda have narrowed the source of the problem.

When I have 1 detector module. All is fine (but I did see few glitches in it in the past). When I put 2 detectors, the first one starts to output garbage like crazy...

I did stop the investigation at that point last night. Lets hope that I'm going to nail the problem soon this morning...

Update:
I think that I have nailed the bug with the help of my friend valgrind!

currently running a test and see how it will go.

Stage 4, day 17:

Last night/this morning, I did run my fifth test.

It did highlight 2 problems. One is a very old issue where I could log garbage when logging the detected profitable trade plans when they are expiring. The reason being that it was possible to update the plan content while it was valid. Actually, you could totally delete it during an update if the update did turn out to result into an unprofitable plan.

I fixed that issue by simply building the new plan in a local variable and swap it with the real deal only if the local one is valid.

Second issue was introduced by my cost rounding optimization idea. I won't go into the details but I'll only say what I have solidified the following lesson:
Cost remains constant for BUY orders, Volume remains constant for SELL orders.

by having fully understood it, the required fix has become obvious

Sixth test planned for later tonight. I start to be eager for the moment where a test will finally go perfectly well... I'm probably very close to that.

When that is going to happen, the next step will be to let the program run for a full 24 hour period. The result will then be the baseline where all my improvement ideas experimentation will compared with.

Funny thing, during the night, I started to write down in a document all the improvements ideas that I had in mind. I can improve that document by changing the order of the items by priority. I must have about a dozen ideas. Each of them makes me enthusiast about doing them. If things are going well, I'm going to love my job in the upcoming months and this is actually a very cool plan/roadmap that is going to create a great financial beast in the crypto trading world... If BASE is helping me acheiving this vision, this is going to be amazing!

Stage 4, day 18:

6th test did happen this afternoon as I was adding some more improvements to the code.

An unseen before now situation did make my code badly behave. 3rd test failing leaving some funds in intermediary alt crypto. I'll need bring all those puppies back home somehow.

The unseen before now situation is now addressed in the code. 7th will either be successful or fail on something else again...

Stage 4, day 19:

I got 2 flawless executions last night. I went to bed very enthusiast that I would have a nice surprise at my awakening this morning.

Alas, it didn't happen. Still more unexpected issues during the last execution that did happen while I was sleeping. I did tweak my logging so that I have the needed info to understand issue in that category easily in the future.

I still have fund stuck in some intermediary currency. However this time, it is USD instead of some low-volume crypto. (I got rid of my Watermelon coins yesterday after a long struggle!)

I wasn't error checking the result of a function and this was leaving a dangling structure in memory. I did address that as well.

I guess that I kept the hardest for the end. I need to understand the core issue that made the whole thing fail. I'm not sure that it will be possible with the amount of info that I currently have. I'll try but it is going to be hard.

Maybe, I will end up resetting everything and hope to have again the same situation repeating with the new logs in place. I prefer not do that because I have no idea how long it can take before it repeats (first time it was happening...). It could take a long long time. I prefer a lot to take care of this problem now and then let it go to restore my full confidence in the system.

I'm a bit frustrated but I stay positive by saying that for every failure that I encounter, it helps me making my system more robust and accurate...

Update:

I have found the problem. It is a nuance between average price and order price that some part of the code didn't make.

I'm not even sure how exactly I'm going to address it yet. I need to contact the exchange support to have clarification of their own rules regarding those 2 prices when receiving orders first. but at least, understanding what did happen is a HUGE relief.

This made me realize one thing. On one hand, I love building this system. It is fun and challenging. OTOH, I now start to be really deep in the rabbit hole and the system complexity has reached a point where getting an understanding of what is going on inside the system requires a good amount of brain juice...

Stage 4, day 21:

I fixed the issue reported in the last journal entry. I did receive some clarification from the exchange support person. Upon receiving this info, this has generated a click in my head.

While I was waiting for their answer, I started to plan very complex possible solutions to the problem. On receiving the reply, this made me figure out a very simple and elegant solution.

I did put it in place. A second also important issue is that for some reason, some updates are never received. The result of this is false positive since you work with stall and invalid data and this is also contributing to less than optimum execution. There are 3 major techniques for execution: joining, improving and lifting the price.

Joining is done by setting your order price to the best current price. Improving is to set the price one notch above the current best price. Lifting is simply to set your price just 1 notch below the opposite side best price. Of course, if you don't have the correct best prices, you cannot execute those strategies correctly.

For high volume pairs, the situation auto correct itself rapidly but it can stay there for quite some time for low volume pairs. This is one factor why I got stuck for so long in MLN the other night.

My plan to address this issue was to subscribe to the spread channels to validate and correct my local trade books data. I was apprehending this change because it is touching a module that I have left alone since last December and it did cause me quite some time to get it right. This is the module that receives about 1 million update per hour and it took me 3 weeks-1 month to make it rock solid and fail proof.

but carefully, I did add the spread update function. I'm quite satisfied of the result as it did give me the opportunity to perform some cleanup/refactoring of that module.

I went to bed after letting this new version running. It did turn out to be quite flawed. It didn't work at all and of course because of that, no trading opportunity got detected.

I did debug the problem this morning. It is quite an art to create good logs without flooding the log file with garbage. It is quite easy to flood the logs when the code receive 1 million updates/hour. The key is to craft logging conditions. You only log when certain conditions are met. This is filtering a lot of good updates and only print out when problems are detected.

Those extra logs takes a lot of room in the log file. I'll keep those extra logs for few hours just to make sure that all is good then I'll deactivate them.

Very happy of current state. 2 major issues got resolved.

I'll let it run and collect the trading execution to monitor them. I'm pretty close to be able to let it run unsupervised for longer periods and I have plenty of ideas for improvement...

but I need to give some attention to my marketing biz that I neglected a bit during last week...

To finish this entry. Here is some stats so far:

The trading volume generated by my tests are getting close to $3000 with a $100 budget. So roughly 30 trades done by my system so far.

I did lose some money but not too much. I had an initial $600 balance. I'm currently at $500 which isn't that much loss considering all the bugs and errors that my tests did encounter.

Stage 4, day 24:

since last update, I did fix the numerous glitches that I did observe the day of the last update.

This has made the system more robust and reliable, running in a predictable way.

I had an euphoric moment when I saw a trade giving a 14% yield in 3 orders executed in less than 1 second. It made roughly $20 profit with a $110 investment. Upon closer inspection, nay. I got this superb yield because I had €13 laying around that got picked up during the execution...

The system is now stable enough for starting a 24h run to use as a baseline. The last 2 trades are making me change my tasks priority. I had a bunch of nice to have or nice to improve ideas but, they will need to wait.

The last 2 trades gave -1.14% and -0.71%. The estimated yield for the last trade was 0.38% or 0.00005 BTC. I lost 0.00009 BTC instead. Very small slippage can change a profitable trade into a loss. I might just increase the bar on the needed yield to jump on the opportunity to fix the issue. but this week, the crypto markets are very quiet. We are maybe in the eye of the storm as everyone is waiting to see what the effect of BTC halving will have on crypto values. I'm saying this because if I had 200 trade opportunities/day, it would have been obvious to raise the bar to favor quality over quantity. This week, it looks more like about a dozen opportunities per day. If I raise too much the bar, there will be no trading at all. Hopefully, I think that this is a transient situation only.

OTOH, I still have an idea or 2 on how to improve the execution efficiency. I'll look into those ideas in priority. As unless I can get this system profitable, there is no point in improving any other aspect of it.

That is how I feel about what needs to be done next. Maybe when I have data for the 24h run. I'll see things differently as the loss could be smoothed out when averaged with the winning trades.

But anyhow, with improving the execution to make more profitable trades, I can't go wrong. So this is what I'm going to work on next.