Crashing with TIR-Learner

[Neato-Nick]

Hi,

I copy and pasted the installation instructions from the README and am running the the script in the active EDTA environment. It seems that the EDTA.pl script chokes trying to use TIR-Learner. Looking at my output, all the correct folders and such are there. After crashing, the Helitron, MITE, and TIR folders are empty but the LTR folder is not. The only file in the parent output folder is genome.fasta.LTR.raw.fa.

Is there a way to run the Perl pipeline script but just not use TIR-Learner, or even just not call TIRs? I'm still interested in the other features, and even if I could just use EDTA for Helitrons, LTRs, MITEs, filtering, consensus calling, and repeat classifying I would be happy.

The lines before the crash start with what's seen in #2 (comment). Then it's a traceback starting from ~/bin/EDTA/bin/TIR-Learner1.12/Module1/Fullcov.py, line 52, in <module> ProcessHomology(genome_Name). After that, there's some cryptic errors including
cat: '*DTA-+-select.fa': No such file or directory
cat: '*-+-*-+-*.gff3': No such file or directory
There's a few more error traces after that, with each Traceback followed by various errors from files not being found by rm, cp, mv, cat.

• TIR-Learner1.12/Module1 (above)
• TIR-Learner1.12/Module1/Lowcomp_M1.py
• TIR-Learner1.12/Module2/Lowcomp_M2.py
• TIR-Learner1.12/

Lastly, in the last few lines before the crash, I get these lines which tell me that it certainly is a problem with TIR-Learner
FileNotFoundError: [Errno 2] No such file or directory: 'TIR-Learner_FinalAnn.gff3' mv: cannot stat 'TIR-Learner/*FinalAnn.gff3': No such file or directory mv: cannot stat 'TIR-Learner/*FinalAnn.fa': No such file or directory cp: cannot stat 'TIR-Learner-Result/TIR-Learner_FinalAnn.fa': No such file or directory Error: TIR results not found!

ERROR: Raw TIR results not found in genome.fasta.EDTA.raw/genome.fasta.TIR.raw.fa at ~bin/EDTA/EDTA.pl line 145.

While bug testing I've just been using the first two scaffolds of my genome. That file is attached.

Thanks!

PR-102_JGI_twoscafs.fasta.zip

[oushujun]

Hi Nick,

We identified this bug in TIR-Learner as you described in detail. A testing version has been pushed in the EDTA branch named "TIR-Learner1.13". Please try that out under the same active EDTA environment (no need to reinstall). In particular, if you want to just test out TIR-Learner, you can:

nohup sh .....EDTA/bin/TIR-Learner1.13/TIR-Learner.sh genome.fa $CPU

For your other question, yes. To do so, you can run these initial TE finders separately, then feed them to the EDTA_process.pl pipeline to make the stage 1 library. If you don't have TIR-Learner results, you can use the MITE-Hunter result to feed the -tir parameter to trick the program.

Please let me know if you still encounter the same issue. Sorry for the inconvenience.

Best,
Shujun

[Neato-Nick]

I pulled from the TIR-Learner 1.13 branch and just ran TIR-Learner as you suggested. Looks like it's still crashing. Module1 did contain some results but there was still a temp file in there, so I'm not sure it finished running. The nohup.out is attached

CPU=4
nohup sh ~/bin/EDTA-TIR-Learner1.13/bin/TIR-Learner1.13/TIR-Learner.sh genome.fasta $CPU

nohup.TIR-Learner1.13.txt

[oushujun]

Hi Nick,

Thanks for testing, @weijiaweijia is working on this. I will update you once we have a new version.

Best,
Shujun

[oushujun]

Hi Nick,

If you are under a pressing need, you may run the updated TIR-Learner1.13 branch for your genome. I just temporarily removed the TIR-Learner module in EDTA, thus you should be able to run the rest of the pipeline. Note that due to the missing of TIR-Learner, large TIR elements and autonomous TIR elements will likely be dampened in the final library. However, the MITE-Hunter should be able to pick up most of short TIR elements and MITEs.

Best,
Shujun

[DanJeffries]

Hi Shujun,

I have an crash that seems similar to Nick's above. Here is the log file:

Wed Jun 26 13:40:03 CEST 2019   Dependency checking:
                All passed!
Wed Jun 26 13:40:14 CEST 2019   Obtain raw TE libraries using various structure-based programs:
FASTA-Reader: Ignoring invalid residues at position(s): On line 380: 5993-6353
FASTA-Reader: Ignoring invalid residues at position(s): On line 534: 225-229
FASTA-Reader: Ignoring invalid residues at position(s): On line 250: 1224-1597, 1622-1750
FASTA-Reader: Ignoring invalid residues at position(s): On line 252: 520-746
FASTA-Reader: Ignoring invalid residues at position(s): On line 386: 238-242, 1481-1485, 2106-2110, 3127-3131
FASTA-Reader: Ignoring invalid residues at position(s): On line 254: 1936-2299
FASTA-Reader: Ignoring invalid residues at position(s): On line 566: 11-15
FASTA-Reader: Ignoring invalid residues at position(s): On line 472: 268-272
Traceback (most recent call last):
  File "/stn4/djeffrie/EDTA/bin/TIR-Learner1.12/Module1/Fullcov.py", line 52, in <module>
    ProcessHomology(genome_Name)
  File "/stn4/djeffrie/EDTA/bin/TIR-Learner1.12/Module1/Fullcov.py", line 41, in ProcessHomology
    f = pd.read_csv(blast, header=None, sep="\t")
  File "/scratch/temporary/djeffrie/EDTAcondaenv/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/scratch/temporary/djeffrie/EDTAcondaenv/lib/python3.7/site-packages/pandas/io/parsers.py", line 429, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/scratch/temporary/djeffrie/EDTAcondaenv/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in __init__
    self._make_engine(self.engine)
  File "/scratch/temporary/djeffrie/EDTAcondaenv/lib/python3.7/site-packages/pandas/io/parsers.py", line 1122, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/scratch/temporary/djeffrie/EDTAcondaenv/lib/python3.7/site-packages/pandas/io/parsers.py", line 1853, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 545, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file
cat: *DTA-+-select.fa: No such file or directory
cat: *DTC-+-select.fa: No such file or directory

The issue again seems to be in TIR-learner. Perhaps it is still a TIR learner bug, but one thing that might be worth noting is that I had an issue during installation where scikit-learn=0.19.0 would not install because of some conflict with multiprocesses. I got around this problem by installing them in a different order but I later realised that by default on my cluster, python 3.7 gets installed in the environment and then a version of multiprocess that is only compatible with python 3.7 is installed. I think then the issue with scikit-learn=0.19.0 was because it only works with python 3.6.

So do you think my issue above could be an installation issue, or a bug in TIR-learner?

One more thing, I am dealing with a large genome of about 5 Gb. The LTR programs completed fine, but took about 3 days. So I was wondering if it is possible to re-use these outputs, rather than waiting another 3 days to see if the pipeline will pass the next step?

Thanks a lot in advance for your help

Best

Dan

[oushujun]

Hi Dan,

Thanks for testing. Yes, this is the issue of TIR-Learner. We are working to make a better version so please wait a week or two.

For the conflicts between python, scikit-learn, and multiprocess, you may try different versions of python and multiprocess, but the trained models do require scikit-learn=0.19.0 to work properly.

The last suggestion is actually on my to-do list. Good idea!

Again, I am sorry for the bugs keeping you from getting meaningful results. We hope to resolve this issue in the near future.

Best,
Shujun

[philippbayer]

In my case with TIR-Learner, my installed CentOS did not have a installed realpath executable which TIR-Learner is calling on line 30 in TIR-Learner.sh

I fixed it like this:

#genomeFile=`realpath $rawFile` #the genome file with real path
genomeFile=`readlink -e $rawFile`

[oushujun]

Thanks @philippbayer!

I did some research and found this multi-platform solution:

resolve_link() {
  if type -p realpath >/dev/null; then
    realpath "$1"
  elif type -p greadlink >/dev/null; then
    greadlink -f "$1"
  else
    readlink -f "$1"
  fi
}

Ref: basherpm/basher#49 (comment)

Changes will be reflected in the next version.

Best,
Shujun

[Neato-Nick]

@oushujun would it be possible to push these changes to a development branch ahead of the release for your next version? Or do you think you are only 1-2 weeks away from your next release?

If my impatience is overwhelming, it might just be easier for me to fix as you and @philippbayer have suggested.

@philippbayer after you made those changes to TIR-Learner, did EDTA run properly?

[philippbayer]

@Neato-Nick The main branch didn't like my GLIBC, so I switched to the origin/TIR-Learner1.13 branch for testing and that one is happily chugging along, but it hasn't finished so far (14 threads, plant genome, still in the 'raw' stage. grf-main and blastall are happily consuming resources, but nothing has been written in a while. The TIR, LTR, and MITE directories have data inside them.

[oushujun]

Hi @Neato-Nick and @philippbayer,

Thank you for waiting patiently, and I am sorry for the prolonged time of development. I went to the Evolution meeting 2 weeks ago so there was some delay there.

I am working on a new version of EDTA, this version will have much better performance in both speed and quality. The main improvement is in TIR-Learner - @weijiaweijia and me are working together to make an improved, more generalized prediction model that fits most species; and also in the downstream filtering of TIR elements and Helitrons - I am working to provide more thorough filtering for raw predictions which will make the final library much smaller and better.

I should be able to push these updates in 1-2 weeks if things work well - - our HPC has been down for maintenance for 3 days, so I can do nothing but talking ...

Again, thank you for your interest and testing.

Best,
Shujun

[philippbayer]

No worries @oushujun :) I'm just playing around with this software, the outcome doesn't depend on anything. Take all the time you need!!

[oushujun]

Dear All,

Sorry for the delay of response. I just push a bulk update to EDTA and have tested it in different servers - it seems to work now. But I have not tested it in macOS, so some tiny differences could cause problems.

For testing purposes, please use a small file, ie. 20 Mb, for faster turn around. Please let me know if there are any issues.

Best,
Shujun