마카로닉스

1575 No

스프링 , 스프링부트 url 간편 스크룰링 및 json 키값 분리 후 간편 파싱 및 자동으로 자바빈객체 넣는 방법

2018-09-18 22:04:05 By 라이트 0 843 댓글수 4 포스트 하기

11월 2024
일	월	화	수	목	금	토
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30

개발 환경은 스프링 부트 1.4.0 버전이다.

json 데이터는 다음과 같다.

https://m.store.naver.com/sogum/api/businesses?filterId=s11556055&filterOpening=true&query=자장명포&start=1001&display=3

{
	"total": 178,
	"nlu": {
		"queryResult": {
			"q": "짜장면",
			"dbKeyword": {
				"isDefault": true,
				"getType": "search",
				"name": "짜장면",
				"viewType": "1col",
				"type": "restaurant",
				"hasComponents": false,
				"useFilter": true
			},
			"location": {
				"default": true,
				"latitude": "37.5666100",
				"x": "126.9783880",
				"y": "37.5666100",
				"longitude": "126.9783880"
			},
			"menu": "짜장면",
			"type": "order",
			"keyword": "짜장면",
			"dbQuery": {
				"isDefault": true,
				"getType": "search",
				"name": "짜장면",
				"viewType": "1col",
				"type": "restaurant",
				"hasComponents": false,
				"useFilter": true
			},
			"showLocationBarFlag": false
		},
		"user": {},
		"deviceInfo": {
			"os": "pc",
			"ie": "0",
			"version": 0
		},
		"queryType": "restaurant"
	},
	"query": {
		"item": [{
			"biz": "짜장면",
			"region_info": {
				"name": "",
				"rcode": ""
			},
			"coordinate_mode": "0",
			"biz_type": "category",
			"rank": "1",
			"region_type": "lba",
			"region_keyword": ""
		}, {
			"biz": "짜장면",
			"region_info": {
				"name": "",
				"rcode": ""
			},
			"coordinate_mode": "0",
			"biz_type": "unknown",
			"rank": "2",
			"region_type": "lba",
			"region_keyword": ""
		}],
		"select": "1",
		"petrol": "0",
		"type": "local",
		"global_menu": "0"
	},
	"smartqueryCoordinates": null,
	"items": [{
		"imageCount": 11,
		"moreUGCReviewsPath": "/restaurants/fsasReviews?id=1618304054&name=%EC%8F%98%ED%95%AB&category=restaurant",
		"distance": "640.37",
		"imageSrc": "http://ldb.phinf.naver.net/20180716_74/1531728211471HsMn5_JPEG/OdNXSXBu2Fc9hq-_SgSMLH19.JPG.jpg",
		"virtualPhone": "0507-1350-4999",
		"totalReviewCount": "13",
		"roadAddr": "서울 종로구 삼일대로17길 47-1 지하1층",
		"options": "예약,단체석,무선 인터넷,포장,남/녀 화장실 구분",
		"priceCategory": "1만원 대",
		"id": "1618304054",
		"addr": "관철동 175",
		"street_panorama": "ijDZ/Gizf3bRDP/yq9Bwfg==,173.87,10.00,126.9849299,37.5692042,120",
		"moreFsasReviewsPath": "/restaurants/fsasReviews?id=1618304054&name=%EC%8F%98%ED%95%AB&category=restaurant",
		"bookingReviewCount": "0",
		"hasBooking": false,
		"dbType": "drt",
		"commonAddr": "서울 종로구",
		"blogCafeReviewCount": "13",
		"routeUrl": "http://m.map.naver.com/viewer/route.nhn?ename=%EC%8F%98%ED%95%AB&ex=126.9849432&ey=37.5690822&edid=1618304054",
		"phone": "010-5704-4999",
		"name": "쏘핫",
		"businessCategory": "restaurant",
		"x": "126.9849432",
		"streetViewUrl": "http://m.map.naver.com/viewer/panorama.nhn?street=on&pinType=site&pinId=1618304054",
		"y": "37.5690822",
		"category": "중식당",
		"desc": ""
	}, {
		"imageCount": 11,
		"moreUGCReviewsPath": "/restaurants/fsasReviews?id=1243240541&name=%EC%86%8C%EA%B3%A0%EC%82%B0%EC%A0%9C%EC%9D%BC%EB%A3%A8%20%EC%B6%A9%EB%AC%B4%EB%A1%9C%EC%A0%90&category=restaurant",
		"distance": "1379.20",
		"imageSrc": "http://ldb.phinf.naver.net/20180730_21/1532942319296qQXTp_JPEG/uHkL5AQZvrhdzDtCpdjXtnpV.JPG.jpg",
		"virtualPhone": "0507-1323-2411",
		"totalReviewCount": "17",
		"roadAddr": "서울 중구 퇴계로30길 14",
		"options": "예약,단체석,주차,발렛파킹,포장,배달,남/녀 화장실 구분,무선 인터넷",
		"priceCategory": "2만원 대",
		"id": "1243240541",
		"addr": "필동1가 24-11",
		"street_panorama": "EOCnTnU6jtA5Usse4FgIvA==,-48.27,10.00,126.9919683,37.5602110,120",
		"moreFsasReviewsPath": "/restaurants/fsasReviews?id=1243240541&name=%EC%86%8C%EA%B3%A0%EC%82%B0%EC%A0%9C%EC%9D%BC%EB%A3%A8%20%EC%B6%A9%EB%AC%B4%EB%A1%9C%EC%A0%90&category=restaurant",
		"bookingReviewCount": "0",
		"hasBooking": false,
		"dbType": "drt",
		"commonAddr": "서울 중구",
		"blogCafeReviewCount": "17",
		"routeUrl": "http://m.map.naver.com/viewer/route.nhn?ename=%EC%86%8C%EA%B3%A0%EC%82%B0%EC%A0%9C%EC%9D%BC%EB%A3%A8%20%EC%B6%A9%EB%AC%B4%EB%A1%9C%EC%A0%90&ex=126.9918536&ey=37.5603134&edid=1243240541",
		"phone": "02-2273-2411",
		"name": "소고산제일루 충무로점",
		"businessCategory": "restaurant",
		"x": "126.9918536",
		"streetViewUrl": "http://m.map.naver.com/viewer/panorama.nhn?street=on&pinType=site&pinId=1243240541",
		"y": "37.5603134",
		"category": "중식당",
		"desc": ""
	}, {
		"imageCount": 5,
		"bookingReviewCount": "0",
		"hasBooking": false,
		"distance": "751.17",
		"dbType": "drt",
		"imageSrc": "http://ldb.phinf.naver.net/20180903_223/1535963652902ll6VG_JPEG/1gMZ_1Qw9I7eeVIY3cT_ofK2.jpeg.jpg",
		"virtualPhone": "0507-1368-5858",
		"commonAddr": "서울 중구",
		"totalReviewCount": "0",
		"roadAddr": "서울 중구 명동2길 49 2층",
		"blogCafeReviewCount": "0",
		"routeUrl": "http://m.map.naver.com/viewer/route.nhn?ename=%EB%A7%88%EB%9D%BC%ED%83%84%ED%83%84%EB%A9%B4&ex=126.9833030&ey=37.5610983&edid=1535545731",
		"phone": "070-7762-5858",
		"name": "마라탄탄면",
		"businessCategory": "restaurant",
		"x": "126.9833030",
		"streetViewUrl": "http://m.map.naver.com/viewer/panorama.nhn?street=on&pinType=site&pinId=1535545731",
		"priceCategory": "2만원 대",
		"y": "37.5610983",
		"id": "1535545731",
		"category": "중식당",
		"addr": "충무로1가 24-18",
		"street_panorama": "c81OnyzleIlgnJylkM2Bnw==,115.71,10.00,126.9832574,37.5611202,120",
		"desc": ""
	}]
}

url 에서 json 을 스크룰링 한 후 데이터에서 item 의 키값에 있는 json 배열 값들만 가져와서 추출해서 가져오는 것이 목표이다.

json 메이븐 의존성 주입

1. 우선 스크룰링을 먼저 하자.

개발후 바로 정리를 했어야 했는데, json 파싱과 섞여 있어 정확히 기억 이 안난다.

의존성 주입한 것은 다음과 같다.

		<!-- https://mvnrepository.com/artifact/org.json/json -->
		<dependency>
			<groupId>org.json</groupId>
			<artifactId>json</artifactId>
		</dependency>

		<dependency>
			<groupId>com.fasterxml.jackson.core</groupId>
			<artifactId>jackson-databind</artifactId>
		</dependency>

		<dependency>
			<groupId>com.fasterxml.jackson.core</groupId>
			<artifactId>jackson-core</artifactId>
		</dependency>

다음 코드가 스크룰링 하는 코드 이다.

package macaronics.net.controller;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;

import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;

import com.fasterxml.jackson.databind.ObjectMapper;

import macaronics.net.domain.NaverFilterVO;
import macaronics.net.domain.NaverVO;

/**
 * @author Choi Jun-Ho(braverokmc79@gmail.com)
   @since 2018. 9. 18. 오후 3:53:21
 */
@Controller
public class NaverGetJsonController {

	private static final Logger log = LoggerFactory.getLogger(NaverGetJsonController.class);

	private static String readAll(Reader rd) throws IOException {
		StringBuilder sb = new StringBuilder();
		int cp;
		while ((cp = rd.read()) != -1) {
			sb.append((char) cp);
		}
		return sb.toString();
	}

	public static JSONObject readJsonFromUrl(String url) throws IOException, JSONException {
		InputStream is = new URL(url).openStream();
		try {
			BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
			String jsonText = readAll(rd);
			JSONObject json = new JSONObject(jsonText);
			return json;
		} finally {
			is.close();
		}
	}

	@ResponseBody
	@RequestMapping("/naver")
	public String getJsonNaver(NaverFilterVO naverFilter) throws Exception {
		naverFilter.setQuery("true");
		log.info("getJson  " + naverFilter.toString());

		String paramQuery = "filterOpening=" + naverFilter.isFilterOpening() + "" + "&query=" + naverFilter.getQuery()
				+ "&start=" + naverFilter.getStart() + "&display=" + naverFilter.getDisplay();

		String url = "https://m.store.naver.com/sogum/api/businesses?" + paramQuery;
		JSONObject json = readJsonFromUrl(url);
      
		return json.toString();
	}
	
	
	
	

	

}

2. 파싱을 하자.

데이터 키값은 다음과 같다.

/*@JsonAutoDetect(fieldVisibility=Visibility.NONE, getterVisibility = Visibility.NONE, setterVisibility = Visibility.NONE)
@JsonPropertyOrder({"unit_id", "unit_code", "lvl", "name", "regDate", "orders", "imgPath", "status"})
public class UnitVO extends CommonVO implements Cloneable {

     @JsonProperty("unitID") @JsonSerialize(using = ToStringSerializer.class) int unit_id;                   
     @JsonProperty("unitCode") String unit_code;              
     @JsonProperty("lvl") @JsonSerialize(using = ToStringSerializer.class) int lvl;                        
     @JsonProperty("name") String name;                   
     @JsonProperty("regDate") String reg_date;              
     @JsonProperty("orders") @JsonSerialize(using = ToStringSerializer.class) int orders;                        
     @JsonProperty("imgPath") String img_path;              
     @JsonProperty("status") @JsonSerialize(using = ToStringSerializer.class) int status;                        
    
     int apistatus;                   
    
     String upper_unit_code;         
     int subcnt;                        
     String statusdesc;              
     String loginid;                   
     */
     /* getter와 setter는 생략 */


/*imageCount: 11,
moreUGCReviewsPath: "/restaurants/fsasReviews?id=1618304054&name=%EC%8F%98%ED%95%AB&category=restaurant",
distance: "640.37",
imageSrc: "http://ldb.phinf.naver.net/20180716_74/1531728211471HsMn5_JPEG/OdNXSXBu2Fc9hq-_SgSMLH19.JPG.jpg",
virtualPhone: "0507-1350-4999",
totalReviewCount: "13",
roadAddr: "서울 종로구 삼일대로17길 47-1 지하1층",
options: "예약,단체석,무선 인터넷,포장,남/녀 화장실 구분",
priceCategory: "1만원 대",
id: "1618304054",
addr: "관철동 175",
street_panorama: "ijDZ/Gizf3bRDP/yq9Bwfg==,173.87,10.00,126.9849299,37.5692042,120",
moreFsasReviewsPath: "/restaurants/fsasReviews?id=1618304054&name=%EC%8F%98%ED%95%AB&category=restaurant",
bookingReviewCount: "0",
hasBooking: false,
dbType: "drt",
commonAddr: "서울 종로구",
blogCafeReviewCount: "13",
routeUrl: "http://m.map.naver.com/viewer/route.nhn?ename=%EC%8F%98%ED%95%AB&ex=126.9849432&ey=37.5690822&edid=1618304054",
phone: "010-5704-4999",
name: "쏘핫",
businessCategory: "restaurant",
x: "126.9849432",
streetViewUrl: "http://m.map.naver.com/viewer/panorama.nhn?street=on&pinType=site&pinId=1618304054",
y: "37.5690822",
category: "중식당",
desc: "
*/
//}

위 json 배열 키값 에 동일한 자바빈 객체를 설정하자.

json 에 자동으로 탐지해서 데이터를 넣어준다.

package macaronics.net.domain;

import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.fasterxml.jackson.databind.ser.std.ToStringSerializer;
/**
 * @author Choi Jun-Ho (braverokmc79@gmail.com)
 */
@JsonAutoDetect
@JsonIgnoreProperties({"query","total", "smartqueryCoordinates", "nlu"})
public class NaverVO implements Cloneable {
	
	
    @JsonSerialize(using = ToStringSerializer.class)
	private int imageCount; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String moreUGCReviewsPath;
    
    @JsonSerialize(using = ToStringSerializer.class)
	private double distance ;		
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String imageSrc ;
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String virtualPhone; 
    
    @JsonSerialize(using = ToStringSerializer.class)    
	private String totalReviewCount ; 
	
    @JsonSerialize(using = ToStringSerializer.class)
    private String roadAddr ;  
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String options; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String priceCategory ;  
    
    @JsonSerialize(using = ToStringSerializer.class)
	private int id;  
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String addr;	
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String street_panorama ; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String  moreFsasReviewsPath ;
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String bookingReviewCount ; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private boolean hasBooking  ; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String dbType; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String commonAddr ; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private int blogCafeReviewCount ; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String routeUrl; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String phone;
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String name; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String businessCategory; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private double x ;
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String streetViewUrl; 
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String y  ;
	
    @JsonSerialize(using = ToStringSerializer.class)
	private String category;
    
    @JsonSerialize(using = ToStringSerializer.class)
	private String desc ;

   
setter, getter

toString()
	
	
}

NaverGetJsonController 전체 소스코드이다.

package macaronics.net.controller;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;

import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;

import com.fasterxml.jackson.databind.ObjectMapper;

import macaronics.net.domain.NaverFilterVO;
import macaronics.net.domain.NaverVO;

/**
 * @author Choi Jun-Ho(braverokmc79@gmail.com)
   @since 2018. 9. 18. 오후 3:53:21
 */
@Controller
public class NaverGetJsonController {

	private static final Logger log = LoggerFactory.getLogger(NaverGetJsonController.class);

	private static String readAll(Reader rd) throws IOException {
		StringBuilder sb = new StringBuilder();
		int cp;
		while ((cp = rd.read()) != -1) {
			sb.append((char) cp);
		}
		return sb.toString();
	}

	public static JSONObject readJsonFromUrl(String url) throws IOException, JSONException {
		InputStream is = new URL(url).openStream();
		try {
			BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
			String jsonText = readAll(rd);
			JSONObject json = new JSONObject(jsonText);
			return json;
		} finally {
			is.close();
		}
	}

	@ResponseBody
	@RequestMapping("/naver")
	public String getJsonNaver(NaverFilterVO naverFilter) throws Exception {
		naverFilter.setQuery("true");
		log.info("getJson  " + naverFilter.toString());

		String paramQuery = "filterOpening=" + naverFilter.isFilterOpening() + "" + "&query=" + naverFilter.getQuery()
				+ "&start=" + naverFilter.getStart() + "&display=" + naverFilter.getDisplay();

		String url = "https://m.store.naver.com/sogum/api/businesses?" + paramQuery;
		JSONObject json = readJsonFromUrl(url);
      
		return json.toString();
	}
	
	
	
	

	@RequestMapping("/")
	public String getJsonNaver2(NaverFilterVO naverFilter, Model model) throws Exception {
		// 검색
		naverFilter.setQuery("짜장면");
		naverFilter.setDisplay(3);
		log.info("getJson  " + naverFilter.toString());

		String paramQuery = "filterOpening=" + naverFilter.isFilterOpening() + "" + "&query=" + naverFilter.getQuery()
				+ "&start=" + naverFilter.getStart() + "&display=" + naverFilter.getDisplay();

		// URL 상에서 json 객체를 가져오기
		String url = "https://m.store.naver.com/sogum/api/businesses?" + paramQuery;
		JSONObject json = readJsonFromUrl(url);

		//json 파싱 시작
		JSONObject jsonObject =new JSONObject(json.toString());
        JSONArray jsonArray =jsonObject.getJSONArray("items"); //키값 items 만 가져오기
        
        //list 객체 생성후 담기
        List<NaverVO> list=new ArrayList<>();

        for(int i=0; i<jsonArray.length(); i++) {
        	JSONObject jsonObject2 =(JSONObject)jsonArray.get(i);
        	
        	//json 을 NaverVO 자동으로 파싱해서 넣는 역할 을 한다.
        	NaverVO naverVO=new ObjectMapper().readValue(jsonObject2.toString(), NaverVO.class);
        	list.add(naverVO);
        }
        
        log.info(" json  :  "+ json.toString());
        
		/*
        for(NaverVO vo : list) {
        	log.info(vo.toString());
        }
        */
        
		return "index";
	}
	
	
	

}

NaverFilterVO 검색 분류 빈 객체를 추가 하였다.

public class NaverFilterVO {

	private boolean filterOpening;
	private String query;
	private int start;
	private int display;
	
	public NaverFilterVO() {
		this.filterOpening=true;
		this.query="";
		this.start=1;
		this.display=1000;
	}
	
	public boolean isFilterOpening() {
		return filterOpening;
	}
	public void setFilterOpening(boolean filterOpening) {
		this.filterOpening = filterOpening;
	}
	public String getQuery() {
		return query;
	}
	public void setQuery(String query) {
		try {
			this.query=URLEncoder.encode(query, "UTF-8");
		} catch (UnsupportedEncodingException e) {
			
			e.printStackTrace();
		}		 
	}
	public int getStart() {
		return start;
	}
	public void setStart(int start) {
		this.start = start;
	}
	public int getDisplay() {
		return display;
	}
	public void setDisplay(int display) {
		this.display = display;
	}
	
	
	@Override
	public String toString() {
		return "NaverFilter [filterOpening=" + filterOpening + ", query=" + query + ", start=" + start + ", display="
				+ display + "]";
	}

	
}

자세한 코드는 나의

https://bitbucket.org/

에 있으니 찾아 보면 된다.

spring