- 32
- 0
the write orc file code is:
public static <T> void write(Class cls, List<T> datas,Properties props) throws Exception{
String path = props.getProperty("localTempFile");
JobConf conf = new JobConf();
FileSystem fs = FileSystem.get(conf);
Path outputPath = new Path(path);
fs.delete(outputPath,true);
OrcSerde serde = new OrcSerde();
StructObjectInspector inspector =
(StructObjectInspector) ObjectInspectorFactory
.getReflectionObjectInspector(cls,
ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
OutputFormat outFormat = new OrcOutputFormat();
RecordWriter writer = outFormat.getRecordWriter(fs, conf, outputPath.toString(),
Reporter.NULL);
for(int i= 0; i < datas.size();i++){
T data = datas.get(i);
System.out.println(String.format("dats:,%s,JSON:%s,", data, JSON.toJSONString(data)));
writer.write(NullWritable.get(), serde.serialize(data,inspector));
}
writer.close(Reporter.NULL);
System.out.println(fs.deleteOnExit(new Path("." + path + ".crc")));
fs.close();
System.out.println("write success .");
}
the test class is:
static class Student {
private String NAME;
private int AGE;
public Student() {
}
public String getNAME() {
return NAME;
}
public void setNAME(String NAME) {
this.NAME = NAME;
}
public int getAGE() {
return AGE;
}
public void setAGE(int AGE) {
this.AGE = AGE;
}
}
the read orc file is:
public static void read(String path) throws IOException, IOException {
Configuration conf = new Configuration();
conf.set("mapreduce.framework.name", "local");
conf.set("fs.defaultFS", "file:///");
Reader reader = OrcFile.createReader(
new Path(path),
OrcFile.readerOptions(conf));
RecordReader records = reader.rows();
Object row = null;
StructObjectInspector inspector
= (StructObjectInspector) reader.getObjectInspector();
List fields = inspector.getAllStructFieldRefs();
for (int i = 0; i < fields.size(); ++i) {
System.out.println("FieldName:" + ((StructField) fields.get(i)).getFieldName() + '\t');
System.out.println("FieldID:" + ((StructField) fields.get(i)).getFieldID() + '\t');
System.out.println("FieldComment:" + ((StructField) fields.get(i)).getFieldComment() + '\t');
System.out.println("TypeName:" + ((StructField) fields.get(i)).
getFieldObjectInspector().getTypeName() + '\t');
System.out.println("name:" + ((StructField) fields.get(i)).
getFieldObjectInspector().getCategory().name() + '\t');
System.out.println("ordinal:" + ((StructField) fields.get(i)).
getFieldObjectInspector().getCategory().ordinal() + '\t');
System.out.println("\n");
}
while (records.hasNext()) {
row = records.next(row);
List value_lst = inspector.getStructFieldsDataAsList(row);
StringBuilder builder = new StringBuilder();
//iterate over the fields
//Also fields can be null if a null was passed as the input field when processing wrote this file
for (Object field : value_lst) {
if (field != null) {
builder.append(field.toString());
}
builder.append('\t');
}
// this writes out the row as it would be if this were a Text tab seperated file
System.out.println(builder.toString()+'\n');
}
}
1
打赏
收藏
点击回答
网站公告
- 扫一扫访问手机版
回答动态

- 神奇的四哥:发布了悬赏问题阿里云幻兽帕鲁服务器更新之后。服务器里面有部分玩家要重新创建角色是怎么回事啊?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题函数计算不同地域的是不能用内网吧?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题ARMS可以创建多个应用嘛?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题在ARMS如何申请加入公测呀?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题前端小程序接入这个arms具体是如何接入监控的,这个init方法在哪里进行添加?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题阿里云幻兽帕鲁服务器刚到期,是不是就不能再导出存档了呢?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题阿里云幻兽帕鲁服务器的游戏版本不兼容 尝试更新怎么解决?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题阿里云幻兽帕鲁服务器服务器升级以后 就链接不上了,怎么办?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题阿里云幻兽帕鲁服务器转移以后服务器进不去了,怎么解决?预计能赚取 0积分收益

- 神奇的四哥:发布了悬赏问题阿里云幻兽帕鲁服务器修改参数后游戏进入不了,是什么情况?预计能赚取 0积分收益
- 回到顶部
- 回到顶部
