Author |
Topic: No fallback plan when batch processing fails (Read 423 times) |
|
AndroYD
Board Newbie
Gender: 
Posts: 16
|
Hello, I'm using the batch export feature, I tried processing 175.495 Musicxml files at first, but it wouldn't even begin to process anything and falsely say the operation was completed, I thought there was probably a limit to how many files H.A. can handle, so after splitting the dataset in two folders of 87.747 files, my deduction proved to be correct, after that it would begin processing the files.
However, when it hits any error, the entire program crashes completely with no chance to resume from where it stopped, this is a big problem, there's no way I can handle this many files manually. Please solve this, I bought H.A. SPECIFICALLY for the batch processing and scripting features, if it had a command line tool like PDFtoMusic Pro or MuseScore I could probably implement my own logic in python for handling such situations, but now I'm feeling completely powerless.
|
|
 offline |
Hello there! My name's Luca! I'm a friendly and empathic person, I alway try to understand the needs of the people around me and create good relationships, I can speak in english, italian, portuguese.
|
|
|
Sylvain Machefert
Administrator
WWW |
Gender:
Posts: 7102
|
 |
Re: No fallback plan when batch processing fails
« Reply #1 on: May 18th, 2021, 11:23am » |
Quote | Modify
|
Hi Luca,
Thousands of files in the same folder is a hard task for OS and disk.
I advise you to split in smaller folders (10000 files), this will speed up processus (not HA dependent) and be more digest for your computer. (and 10000 is still HUGE!)
The best way is to start with small, and increase number of files if everything is OK.
Try with 1000 files.
To find the failed file, you have 2 ways:
- if there is a crash in one of 1000 files, split it in 2. If only one file causes crash, you have a batch of 500 OK, and a batch of 500 fail. Split the last one in 2 and repeat.
- If you script in Python or whatever, after a crash, remove the .musicxml files from the "source" folder if there is a corresponding .myr in the "destination" folder.
This way, you don't re-import the well converted files, you move the first .musicxml file in a "probably-bug" folder.
Once finished, retry manually each file in "probably-bug" folder. If they seem to work, then do a batch conversion. The crash was probably caused by HA's indigestion.
|
|
offline |
HA+HQ+VS+PdfToM, Reaper+Audiveris+Transcribe, Win10+Focusrite Scarlet 4i4+Nord Electro 3
Linktree: VS languages, my scripts, my bands, my performances...
|
|
|
AndroYD
Board Newbie
Gender:
Posts: 16
|
|
Re: No fallback plan when batch processing fails  loopbug1.png
« Reply #2 on: May 27th, 2021, 8:46am » |
Quote | Modify
|
Hello, I found a new bug where H.A. gets stuck in an infinite loop during a batch export and no error messages are displayed.
To reproduce the bug, open any of these files https://github.com/AndroYD84/Files/tree/master/HAproblem3 and try to export them in another format, as soon as you try it will display this message "There is too many played bars in your document. May be there is a problem with the breaks symbols. Do you want to see a summary of the breaks?" and won't let you save it until you have fixed that, I suspect the batch export feature gets stuck because it keeps trying to save these file but H.A. won't allow it.
I'm legit frustrated now because after implementing all the logic suggested above and even creating a python script that can detect any error messages from H.A., isolate the bad files from the good ones, restart the program after crashing and resume operations automatically by controlling my mouse and interacting with the GUI (made with https://github.com/asweigart/pyautogui ), since my script is triggered by error messages now it's useless against it and got to rewrite everything again.
|
|
offline |
Hello there! My name's Luca! I'm a friendly and empathic person, I alway try to understand the needs of the people around me and create good relationships, I can speak in english, italian, portuguese.
|
|
|
Sylvain Machefert
Administrator
WWW |
Gender:
Posts: 7102
|
|
Re: No fallback plan when batch processing fails
« Reply #3 on: May 27th, 2021, 9:43am » |
Quote | Modify
|
Please write to info@myriad-online.com as you found a bug and we, on the forum, can only test and say "ok, we have the same bug too".
Myriad devs don't read all posts here, it's better to contact them by email when you have a bug you can reproduce with details on how you reproduce it.
|
|
offline |
HA+HQ+VS+PdfToM, Reaper+Audiveris+Transcribe, Win10+Focusrite Scarlet 4i4+Nord Electro 3
Linktree: VS languages, my scripts, my bands, my performances...
|
|
|
|
