taofen8 / salyut Goto Github PK
View Code? Open in Web Editor NEWThis is an engine of a spider markup-language which called trico. (spider framework)
Home Page: https://trico.cloud
This is an engine of a spider markup-language which called trico. (spider framework)
Home Page: https://trico.cloud
如下代码
- segment:
package: '"taobaoSearch"'
name: '"pailitaoSearchPic"'
args: { 0: '/url',1: '/isTest',2: '/img',3: '/buyingId' }
body:
- loop:
in: { eles: '#imgsearch-itemlist .item',mutable: '$false' }
each:
使用find标签时,当元素display为none时,也能被找到
# 以下是一个打开页面的segment, 请注意以下注释的注意点
- segment:
name: '"new_animation_update_time"'
package: '"Kiva"'
args: {0: '/url'}
body:
- load: '$/url'
- loop:
in: { eles: '.tl-day' }
each:
- select: { ele: '$e', attr: '"class"', path: '/clazz' }
- if: '$/clazz == "tl-day today"'
- then:
- put: { path: '/todayIndex', value: '$i', lambda: 'x -> return Number(x)' }
- break:
- echo: '"todayIndex: " + $/todayIndex'
- loop:
in: { eles: '.season-timeline' }
each:
- echo: '"i:" + $i'
- echo: '$/todayIndex'
- if: '$i < $/todayIndex'
- then:
- continue:
- put: { path: '/index', value: '$i'}
- echo: '"else:" + $/index'
- select: { eles: '.season-group:not(.season-timer)', under: '$e', path: '/groupList' }
# - loop:
# in: { values: '$/groupList', mutable: '$false' }
# each:
# - select: { ele: '.group-time', under: '$e', path: '/time' }
# - select: { eles: '.season-item', under: '$e', path: '/items' }
# - put: { path: '/head', value: '$/headList[$/index]'}
# - select: { ele: '.t-date', under: '$/head', path: '/date' }
# - select: { ele: '.t-week', under: '$/head', path: '/week' }
# - echo: '$/date'
# - loop:
# in: { values: '$/items', mutable: '$false' }
# each:
# - select: { ele: 'img', attr: 'src', under: '$v', path: '/item/pic' }
# - select: { ele: '.season-title', under: '$v', path: '/item/title' }
# - select: { ele: '.season-desc', under: '$v', path: '/item/desc' }
# - put: { path: '/item/date', value: '$/date' }
# - put: { path: '/item/week', value: '$/week'}
# - put: { path: '/resList[-1]', value: '$/item' }
- return: '$/resList'
- segment:
name: '"__test__new_animation_update_time"'
package: '"Kiva"'
body:
- callin: {seg: '"new_animation_update_time"', 0: '"https://www.bilibili.com/anime/timeline"', package: '"Kiva"'}
- echo: '$1'
# 在trico后台要提交segment的时候,请将以下这一行的callin注释掉。
- callin: {seg: '"__test__new_animation_update_time"', package: '"Kiva"'}
- segment:
name: '"__test__baidunews"'
args: {}
package: '"tom"'
body:
- load: '"http://news.baidu.com/"'
- segment:
name: '"__test__baidunews"'
args:
package: '"tom"'
body:
- load: '"http://news.baidu.com/"'
private static final String ATTRIBUTE_FORMAT = "%s: ";
else if (KeyType.ATTRIBUTE.equals(keyType)){
builder.append(String.format(ATTRIBUTE_FORMAT,tuple2.f0));
}
如下语句报错
- if: '7 > 6.1'
当被click元素display为none时,浏览器直接crash,然后又重启
Please try to add the token distinct, function description is below.
SYNOPSIS
- distinct: { path: , value: }
DESCRIPTION
Distinct the values in the list
ATTRIBUTES
name | type | required |
---|---|---|
path | path | √ |
value | expr | √ |
EXAMPLE
Input: [1,1,2,2,3]
Output: [1,2,3]
- distinct: { path: '/distinct', value: '$/someInputArr' }
Salyut.setEnv(EnvKey.DRIVER_PATH,{your driver path});
Salyut.setEnv(EnvKey.NEW_TAB_PATH,{your newTab file path});
Salyut.launch();
Salyut.setOutputListener(
msg -> {
//引擎处理过程中产生的消息
}
);
Salyut.setResultListener(
result -> {
//返回结果
}
);
String script = {your script content};
TricoScript script = new TricoScript(yaml);
Salyut.execScript(script.getContent(),"","");
String script = {your script content};
TricoScript script = new TricoScript(yaml);
这里是否笔误?
under建议也改用支持变量式的写法 "'dd' + $/v" 除了能使用css选择器字符串外也支持WebElement对象
下面那个loop用了start参数 按照输出$/startIndex为6 但是还是从0开始 之后超出数组界限报错
# 以下是一个打开页面的segment, 请注意以下注释的注意点
- segment:
name: '"new_animation_update_time"'
package: '"Kiva"'
args: {0: '/url'}
body:
- load: '$/url'
- loop:
in: { eles: '.tl-day' }
each:
- select: { ele: '.t-date', parent: '$e', path: '/date' }
- select: { ele: '.t-week', parent: '$e', path: '/week' }
- put: { path: '/time/date', value: '$/date' }
- put: { path: '/time/week', value: '$/week' }
- put: { path: '/list[$i]', value: '$/time' }
- remove: { path: '/time' }
- select: { target: '$e', attr: '"class"', path: '/tlDayClazz' }
- if: '$/tlDayClazz == "tl-day today"'
- then:
- put: { path: '/startIndex', value: '$i' }
- loop:
in: { eles: '.tl-day' }
each:
- if: '$i >= $/startIndex'
- then:
- put: { path: '/res[-1]', value: '$/list[$i]' }
- echo: '$/res'
- echo: '$/startIndex'
- loop:
in: { eles: '.season-timeline', start: '$/startIndex' }
each:
- echo: '$i'
- select: { ele: '$e', path: '/now' }
- echo: '$/now'
- remove: { path: '/now' }
- loop:
in: { eles: '.season-group:not(.season-timer)', parent: '$e' }
each:
- loop:
in: { eles: '.season-item', parent: '$e' }
each:
- select: { ele: 'img', attr: '"src"', parent: '$e', path: '/item/imgSrc' }
- select: { ele: '.season-title', parent: '$e', path: '/item/title' }
- select: { ele: '.season-desc', parent: '$e', path: '/item/desc' }
- put: { path: '/animationList[-1]', value: '$/item' }
- remove: { path: '/item' }
- put: { path: '/now', value: '$/res[$i]' }
- put: { path: '/now/animationList', value: '$/animationList' }
- remove: { path: '/animationList' }
- remove: { path: '/now' }
- segment:
name: '"__test__new_animation_update_time"'
package: '"Kiva"'
body:
- callin: {seg: '"new_animation_update_time"', 0: '"https://www.bilibili.com/anime/timeline"', package: '"Kiva"'}
- echo: '$1'
# 在trico后台要提交segment的时候,请将以下这一行的callin注释掉。
- callin: {seg: '"__test__new_animation_update_time"', package: '"Kiva"'}
目前使用lambda转,可以提供字符串和数字互转的token
- put: {path: '/number' , value: '$/str', lambda: 'x -> return Number(x)+0.01;'}
callin标签中key与value之间没有空格,报空指针,kernel没有抛出异常,导致IDE一直处于running状态。卡住的假象。
示例代码(0:'"test"'之前没有空格)
- segment:
# 特别注意testSegment的入口名必须是以“__test__”开头,(您也可以写多个Segment通过其中callin来进行调用)
name: '"__test__appEdu"'
package: '"school"'
args: {}
body:
- callin: {seg: '"appEdu"', package: '"school"', 0:'"test"', 1: '"https://www.app.edu.gov.on.ca/eng/sift/schoolProfile.asp?SCH_NUMBER=000779"', 2: '"000779"', 3: '"Elementary"' }
- put: {path: '/test', value: '$1'}
- echo: '$/test'
- if: '$/test != $null'
- then:
- return: '$true'
- else:
- return: '$false'
kernel报错信息
1589967087774 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/43/6g6jmkbj2qg6zy1cn_pqqmww0000gp/T/rust_mozprofileLq211j"
1589967088115 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: networkStatus
1589967088325 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: mozillaAddons
1589967088325 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: telemetry
1589967088325 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: resource://pdf.js/
1589967088325 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: about:reader*
1589967090030 Marionette INFO Listening on port 61627
1589967090137 Marionette WARN TLS certificate errors will be ignored for this session
五月 20, 2020 5:31:30 下午 org.openqa.selenium.remote.ProtocolHandshake createSession
信息: Detected dialect: W3C
2020-05-20 17:31:31,618 [release] - java.lang.NullPointerException
when I user switchtab in my segment, I find that switchtab isn't work. I can't understard "index" .
This is my segment . please tell me how to find my tab by index.
- segment:
name: '"calc"'
args: {0: '/input'}
body:
#load page of Baidu
- load: '"https://my.xiapibuy.com/Cushions-Covers-cat.26.681.9888?page=0&sortBy=sales"'
- loop:
in: {eles: 'div.shopee-search-item-result__items div.shopee-search-item-result__item'}
each:
- select: {path: '/tmp/url',under: '$e',ele: 'a[data-sqe=link]',attr: '"href"'}
- put: {path: '/urls',value: '"window.open(\"" + $/tmp/url + "\")"'}
- put: {path: '/tmp/count',value: '$i'}
- echo: '$/urls'
- js: '$/urls'
- wait: {type: '"time"',millis: '4000'}
- switchtemptab: {index: '1'}
- wait: {type: '"presence"',ele: 'button.btn-solid-primary',timeout: '10000'}
- find: {ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card> div.flex.flex-auto > div > div > div:nth-child(1) > div'}
- if: '$1'
- then:
- select: {ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card> div.flex.flex-auto > div > div > div:nth-child(1) > div',path: '/tmp/rank'}
- contains: {path: '/tmp/rank',search: '"RM"'}
- if: '$1'
- then:
- put: {path: '/tmp/rank',value: '"无评分"'}
- else:
- put: {path: '/tmp/rank',value: '"无评分"'}
- find: {ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card> div.flex.flex-auto > div > div > div:nth-child(3) > div:nth-child(1)'}
- if: '$1'
- then:
- select: {path: '/tmp/sale',ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card> div.flex.flex-auto > div > div > div:nth-child(3) > div:nth-child(1)'}
- else:
- put: {path: '/tmp/sale',value: '"无"'}
- find: {ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card> div.flex.flex-auto > div > div > div:nth-child(2) > div:nth-child(1)'}
- if: '$1'
- then:
- select: {path: '/tmp/rant',ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card> div.flex.flex-auto > div > div > div:nth-child(2) > div:nth-child(1)'}
- else:
- put: {path: '/tmp/rant',value: '"无"'}
- find: {ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card > div.flex.flex-auto > div > div:nth-child(3) > div > div > div > div > div.flex.items-center > div:nth-child(1)'}
- if: '$1'
- then:
- select: {path: '/tmp/price',ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card > div.flex.flex-auto > div > div:nth-child(3) > div > div > div > div > div.flex.items-center > div:nth-child(1)'}
- else:
- put: {path: '/tmp/price',value: '"无售价"'}
- find: {ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card > div.flex.flex-auto > div > div > span'}
- if: '$1'
- then:
- select: {path: '/tmp/title',ele: '#main > div > div.shopee-page-wrapper > div.page-product > div.container > div.product-briefing.flex.card > div.flex.flex-auto > div > div > span'}
- else:
- put: {path: '/tmp/title',value: '"未获取到"'}
- put: {path: '/R/list[-1]',value: '$/tmp'}
- remove: {path: '/tmp'}
- closetemptab:
- export: {type: '"excel"',path: '/R/list/',value: 'count,title,price,sale,rank,rant,url'}
- return: '$/R'
#call the segment named 'calc' and pass the argument '1+1'
- callin: {seg: '"calc"', 0: '1+1'}
- echo: '$1'
建议js支持执行完通过return返回一个String类型的值 js改成类似 - js: { path:'/res', str: '"var a=1; return a;"' } 的写法 将a的值保存在变量path内
- if: '$/isTest == "test"'
- then:
- return: '"true"'
- else:
- then:
- return: '"false"'
sample中的webCalc.tr 脚本有语法错误
name前面有6个空格占位,formatYaml此方法中, builder.append(lines[i]); 去掉lines[i]原有空格
private static String formatYaml(String yamlString){
StringBuilder builder = new StringBuilder();
String[] lines = yamlString.split("\n");
builder.append(lines[0]);
builder.append("\n");
for(int i = 1; i < lines.length; i++){
builder.append(" ");
builder.append(lines[i]);
builder.append("\n");
}
return builder.toString();
}
STANDARD_IDENT 应该为4个空格占位,以下代码中只有3个占位。
private static final String STANDARD_IDENT = " ";
最后结果
- segment:
name: '"__test__baidunews"'
args:
package: '"tom"'
body:
- load: '"http://news.baidu.com/"'
- select: { eles: '.ulist li',path: '/arr' }
- len: '$/arr'
- if: '$1 > 0'
- then:
- return: '$true'
- else:
- return: '$false'
private static final String STANDARD_IDENT = " ";
错误语法示例
- if: '($i+1)%6==0'
- then:
- js: '"document.documentElement.scrollTop = 438;"'
- wait: { type: '"time"', millis: '1500'}
- else:
- then:
- echo: '"is wrong!!!"'
salyut exception
2020-04-08 23:54:35,645 [release] - java.lang.NullPointerException
建议ele、eles属性也改用和其他一样的可使用变量式写法 如:‘”dfs“ +
使用mobile标签后运行脚本出现不停重启浏览器,如
- segment:
name: '"taobaoItemDescTest"'
package: '"aichuanda"'
args: { 0: '/test' }
body:
- mobile:
- load: '"https://main.m.taobao.com/index.html"'
- put: {path: '/test', value: '"1234567"'}
- replace: { path: '/test', regex:'"7"', replacement: '"8"' }
segment 参数一直保留着第一次传进去的值
部分示例如下:
- segment:
package: '"school"'
name: '"getReportCardData"'
args: {}
body:
- loop:
in: {eles: '.row table#insidetable2', mutable: '$false'}
each:
- select: { ele: 'caption', under: '$e', path: '/indexType'}
- echo: '$/indexType'
- put: { path: '/index', value: '$i'}
- js: '""'
- callin: {package: '"school"', seg: '"getTableData"', 0: '$/indexType', 1: '$/index'}
- put: {path: '/cardTableList', value: '$1'}
- if: '$/cardTableList != $null'
- then:
- loop:
in: { values: '$/cardTableList'}
each:
- copy: { path: '/reportCardList[-1]', value: '$v'} # $V循环非web element元素的数组元素值
- return: '$/reportCardList'
- segment:
package: '"school"'
name: '"getTableData"'
args: { 0: '/indexType', 1: '/index'}
body:
- echo: '"indexType="+$/indexType'
- echo: '"getTableData="+$/index'
- if: '$/index == 0'
- then:
- loop:
in: { eles: 'tbody#tbody_0 tr', mutable: '$false'}
each:
- if: '$i>1'
- then:
- put: { path: '/tableRow/indexType', value: '$/indexType'}
- select: { path: '/tableRow/indexDesc', ele: 'td:nth-child(1)', under: '$e'}
- select: { path: '/tableRow/school', ele: 'td:nth-child(2)', under: '$e'}
- select: { path: '/tableRow/englishStudents', ele: 'td:nth-child(3)', under: '$e'}
- select: { path: '/tableRow/frenchStudents', ele: 'td:nth-child(4)', under: '$e'}
- put: { path: '/tableList[-1]', value: '$/tableRow'}
- remove: { path: '/tableRow'}
- return: '$/tableList'
- if: '$/index == 1'
- then:
- loop:
in: { eles: 'tbody#tbody_1 tr', mutable: '$false'}
each:
- if: '$i>1'
- then:
- put: { path: '/tableRow/indexType', value: '$/indexType'}
- select: { path: '/tableRow/indexDesc', ele: 'td:nth-child(1)', under: '$e'}
- select: { path: '/tableRow/school', ele: 'td:nth-child(2)', under: '$e'}
- select: { path: '/tableRow/englishStudents', ele: 'td:nth-child(3)', under: '$e'}
- select: { path: '/tableRow/frenchStudents', ele: 'td:nth-child(4)', under: '$e'}
- put: { path: '/tableList[-1]', value: '$/tableRow'}
- remove: { path: '/tableRow'}
- return: '$/tableList'
- if: '$/index == 2'
- then:
- loop:
in: { eles: 'tbody#tbody_2 tr', mutable: '$false'}
each:
- if: '$i>1'
- then:
- put: { path: '/tableRow/indexType', value: '$/indexType'}
- select: { path: '/tableRow/indexDesc', ele: 'td:nth-child(1)', under: '$e'}
- select: { path: '/tableRow/school', ele: 'td:nth-child(2)', under: '$e'}
- select: { path: '/tableRow/englishStudents', ele: 'td:nth-child(3)', under: '$e'}
- select: { path: '/tableRow/frenchStudents', ele: 'td:nth-child(3)', under: '$e'}
- put: { path: '/tableList[-1]', value: '$/tableRow'}
- remove: { path: '/tableRow'}
- return: '$/tableList'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.