账号密码登录
微信安全登录
微信扫描二维码登录

登录后绑定QQ、微信即可实现信息互通

手机验证码登录
找回密码返回
邮箱找回 手机找回
注册账号返回
其他登录方式
分享
  • 收藏
    X
    为什么我读出的orc文件的fileName都是小写
    32
    0

    the write orc file code is:

    public static <T> void write(Class cls, List<T> datas,Properties props)  throws Exception{
    
        String path = props.getProperty("localTempFile");
    
        JobConf conf = new JobConf();
        FileSystem fs = FileSystem.get(conf);
        Path outputPath = new Path(path);
    
        fs.delete(outputPath,true);
    
        OrcSerde serde = new OrcSerde();
       
        StructObjectInspector inspector =
                (StructObjectInspector) ObjectInspectorFactory
                        .getReflectionObjectInspector(cls,
                                ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
        OutputFormat outFormat = new OrcOutputFormat();
    
        RecordWriter writer = outFormat.getRecordWriter(fs, conf, outputPath.toString(),
                Reporter.NULL);
    
        for(int i= 0; i < datas.size();i++){
            T data = datas.get(i);
            System.out.println(String.format("dats:,%s,JSON:%s,", data, JSON.toJSONString(data)));
            writer.write(NullWritable.get(), serde.serialize(data,inspector));
        }
        writer.close(Reporter.NULL);
    
        System.out.println(fs.deleteOnExit(new Path("." + path + ".crc")));
        fs.close();
        System.out.println("write success .");
    
    }
    
    

    the test class is:

    static class Student {
        private String NAME;
        private int AGE;
    
        public Student() {
        }
    
        public String getNAME() {
            return NAME;
        }
    
        public void setNAME(String NAME) {
            this.NAME = NAME;
        }
    
        public int getAGE() {
            return AGE;
        }
    
        public void setAGE(int AGE) {
            this.AGE = AGE;
        }
    }
    
    

    the read orc file is:

     public static void read(String path) throws IOException, IOException {
        Configuration conf = new Configuration();
        conf.set("mapreduce.framework.name", "local");
        conf.set("fs.defaultFS", "file:///");
    
        Reader reader = OrcFile.createReader(
                new Path(path),
                OrcFile.readerOptions(conf));
    
        RecordReader records = reader.rows();
    
        Object row = null;
    
        StructObjectInspector inspector
                = (StructObjectInspector) reader.getObjectInspector();
    
        List fields = inspector.getAllStructFieldRefs();
    
        for (int i = 0; i < fields.size(); ++i) {
            System.out.println("FieldName:" + ((StructField) fields.get(i)).getFieldName() + '\t');
            System.out.println("FieldID:" + ((StructField) fields.get(i)).getFieldID() + '\t');
            System.out.println("FieldComment:" + ((StructField) fields.get(i)).getFieldComment() + '\t');
            System.out.println("TypeName:" + ((StructField) fields.get(i)).
                    getFieldObjectInspector().getTypeName() + '\t');
            System.out.println("name:" + ((StructField) fields.get(i)).
                    getFieldObjectInspector().getCategory().name() + '\t');
            System.out.println("ordinal:" + ((StructField) fields.get(i)).
                    getFieldObjectInspector().getCategory().ordinal() + '\t');
            System.out.println("\n");
        }
    
        while (records.hasNext()) {
            row = records.next(row);
            List value_lst = inspector.getStructFieldsDataAsList(row);
            StringBuilder builder = new StringBuilder();
            //iterate over the fields
            //Also fields can be null if a null was passed as the input field when processing wrote this file
            for (Object field : value_lst) {
                if (field != null) {
                    builder.append(field.toString());
                }
                builder.append('\t');
            }
            //    this writes out the row as it would be if this were a Text tab seperated file
            System.out.println(builder.toString()+'\n');
        }
    
    }
    
    
    1
    打赏
    收藏
    点击回答
        全部回答
    • 0
    更多回答
    扫一扫访问手机版
    • 回到顶部
    • 回到顶部