最近又要开始做些机器翻译方面的实验,于是按wmt08的介绍训练相关的模型,这些去年都做过,这次只是换用wmt08的提供的欧洲平行语料库做相关实验,却发现在利用moses的训练脚本train-factored-phrase-model.perl训练模型时出错
  命令:bin/moses-scripts/scripts-20090113-1019/training/
train-factored-phrase-model.perl -scripts-root-dir bin/moses-scripts/scripts-20090113-1019 -root-dir workdir/baseline -corpus workdir/corpus/europarl.lowercased -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:workdir/lm/europarl.3lm:0 –parallel
  报错提示:“ ERROR: Filename is not absolute: 090211workdir/lm/europarl.3lm at bin/moses-scripts/scripts-20090113-1019/training/
train-factored-phrase-model.perl line 219.”
  从提示来看,指得是语言模型文件路径不是绝对路径,这个问题让我很奇怪,去年训练时是不存在这个问题的,打开这个脚本,将219,220行用#注释掉,然后重新训练,果然这个问题得到解决,可以正常训练相关模型了!
  但是心中仍存疑虑,找来去年2月份moses的版本,打开这个脚本看了一下,发现果然没有那两行注释掉的,为什么moses训练脚本以前支持相对路径反而现在不支持了,这不是将简单的事情搞复杂吗?wmt08的介绍还是用的是相对路径,莫非这是moses脚本的一个bug?于是继续在google中找这个问题的答案,我相信有人会遇到同样的问题,如果他在wmt08的介绍下作实验。不过这次直接找到了这个问题的起因,来自于moses邮件组,在08年11月20日的一封邮件中,这样写道:

  Two more patches attached (I'm still operating under the assumption that this is the best place to send them).
  These patches add an absolute file name check. This bit me when I tried to learn how to use tuning. The train-factored-model script was perfectly happy to let me use relative filenames for the language model. But when I want to do tuning via mert-moses, I got a complaint about the same filename in my autogenerated moses.ini file being non absolute.
  Note that the mert-moses patch is only (presumably) for portability.

  看来这个用户在通过mert-moses调参时发现了相对路径的不方便,于是在训练脚本中加了几行判断绝对路径的语句,并且得到了moses社区的认可,所以现在使用这个脚本时必须在最好调用srilm生成的语言模型的那一行使用绝对路径文件名。
  总而言之,moses训练脚本这个地方的修改虽然死板些,但是可以让全套训练更加自动化,减去人工干预,也不失为一个好办法!

注:原创文章,转载请注明出处“我爱自然语言处理”:www.52nlp.cn

本文链接地址:
https://www.52nlp.cn/one-problem-on-moses-training-scripts/

作者 52nlp

《Moses训练脚本的一个问题》有5条评论
  1. hello! I encountered a problem during training in step 3 with the
    following message

    Using SCRIPTS_ROOTDIR: --corpus
    Using single-thread GIZA
    ERROR: use --corpus to specify corpus at /root/mtworkdir/bin/scripts/scripts-20110321-0844/training/train-model.perl line 212.

    root@ubuntu:~/mtworkdir/www# /root/mtworkdir/bin/scripts/scripts-20110321-0844/training/train-model.perl --scripts-root-dir $SCRIPS_ROOTDIR --corpus /root/mtworkdir/www/corpus/train-6000 --f mg --e ch --alignment grow-diag-final-and-reordering msd-bidirection-fe --lm 0:3:/root/mtworkdir/www/lm/chinese-lm.txt >& /root/mtworkdir/www/training.out &
    [1] 2880

    Has anyone encountered this message before?
    And does anyone have a clue what it means?

    [回复]

    52nlp 回复:

    Sorry, I don't know!

    [回复]

    tianliang 回复:

    you should give the path of the $SCRIPS_ROOTDIR first,so you can run" export SCRIPTS_ROOTDIR=/home/tianliang/research/moses/scripts/target/scripts-20101111-1130" first,
    and then run your program again,have a try!

    [回复]

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注